I am creating an application for a client that can import data from a partner through an API and parse it, determine if the entry being parsed already exists in the Corvid Collections, if yes update, then save.
Simple enough seemingly. Up to now I have managed to connect to the API and have created the corresponding tables and even managed to run the script to fill the base collections.
The parsing script I have created is very simple, get list of items from the partner’s getList API, and then I loop each item, first check if the entry exists in the collection (to determine if save or update), call getItemDetails from partner’s API and save to Wix Collection. It worked, but I noticed that the first time not all items were logged in the collection, the loop didn’t finish. So I started it again and got all the entries.
But the problems really began when I tried to add the “if exists then update”. Of course, I had to use lots of nested promises, but it seems that when there are too much of these nested promises, the code times out, and stops in the middle of execution.
The question I have for you Wix experts is, are these normal problems for applications written in asynchronous javascript, and I am asking too much of it? Or this has been done many times before and can work?
I know in PHP, this script would have been fully finished and working perfectly a few days ago, so it’s very frustrating. Thanks in advance for any insight on this.
Nested promises in JS will always require extra care to ensure proper error handling.
I’d recommend you look into other JS asynchronous methods so that you don’t end up with a bunch of nested promises. Callbacks are still imho the best way to ensure events fire optimally, but you also have async await and possible timeout retry methods.
Also keep in mind JS has quite a few different looping methods and it may be worth it to try different ones to see which best suits your needs.
I have already tried all possibilities. Async, Await, .then in all possible combinations. And a long time ago I experimented with timers for a similar task, and I can tell you this: it doesn’t work, if you want to become crazy insert timers in loops.
I have also tried different kinds of loops, for, while, foreach and in all cases, my loops work perfectly as long as I don’t insert any external API calls in them, then everything goes bullocks, and not always the same way, so that points to timeouts. At first I though Async Await would be the perfect tool for that, but the loop continues even with them.
I have setup the same loops and API calls in PHP, took me a few hours and everything works out of the box. It would have been better to go directly to Wix, but it seems I have simply pushed Wix Corvix beyond it’s capabilities and after a few days of research and tests and no results, I will have to tell my client we must go to the PHP solution as a bridge between our partner’s data and Wix.
Thanks again, you have done your best and it’s much appreciated, but I was hoping someone would say: Hey, I have done this, and I know it can work, here’s exactly how I did it. But it seems it won’t happen.
You could try to debug it with a loop that only continues after some promise has been resolved. If you’re getting a promise pending, then this is especially useful.
let i = 0,
arr = await getList();
loop(arr);
function loop(parseArr) {
iteration(parseArr[i], () => i++);
if (i < arr.length) loop(arr);
}
function iteration(item, callback) {
try {
myFunc(item);
} catch (e) {
console.warn(e);
} finally {
callback();
}
}