[SOLVED] Web timeout ! Query refresh collection: what's wrong?

PLease, help me to solve the rebus.

I have a collection ‘Soci’, the records are the members of an association.
Each member have a code: tessera.
After some time you need to refresh ‘Soci’ collection with new members.

So via db_manager I load the new .csv in ‘importSoci’ with 500 or 1000 records, then I run this routine in order to update ‘Soci’.

The code seems good, BUT when I run the code I get in console:
« #GatewayTimeout » or «WebMethod request #timedout after 14 seconds».

‘Soci’ collection rise from eg. 200 members until to 300 members, then it stops.
Why ?
The routine is in the backend.

thanks in advance
Mauro

import wixData from 'wix-data';

export async function importaSoci() {

let options     = {suppressAuth:true, suppressHooks:true}
let arraySoci =[];

// create 'arraySoci' from 'importSoci' collection
await importaArray(options).then (res => {arraySoci=res});

// FOR: search for the same ‘tessera’ into ‘Soci' collection
for (let i=0; i<arraySoci.length; i++) {

await cercaTessera(arraySoci[i].tessera, options)
.then ( async res => { 
  // if don’t find the same ‘tessera’ insert new record
  if(res==null){
  await insertSoci(arraySoci[i], options);
  } else {
  arraySoci[i]._id = res._id;
  await updateSoci(arraySoci[i], options)
  }
})
.then(() => {erase("importSoci")})
.catch(err => {return err})
    } // CLOSE FOR LOOP

return "REFRESH OK"
} 

//------  FUNCTIONS ________
async function importaArray (op) {
return await wixData.query("importSoci", op)
.limit(1000)
.find()
.then( (results) => { 
return results.items } )
.catch( err => {return err})
}

async function cercaTessera(tessera, op) {
return await wixData.query("Soci", op)
.limit(1000)
.eq("tessera", tessera)
.find()
.then( (results) => { 
return results.items[0] } )
.catch( err => {return err})
}

async function updateSoci(record,op) {
return await wixData.update("Soci", record, op)
.then( (res) => {return res.items})
.catch(err => {return err})
}

async function insertSoci(record, op) {
return await wixData.insert("Soci", record, op)
.then( (res) => {return res.items})
.catch(err => {return err})
}

That WebMethod error normally means that the WebMethod sometimes takes some time to calculate and it is possible that the timeout for the application is reached before the WebMethod has finished.

Whilst Wix have their own support page for 502 Bad Gateway error, which gives a similar reply as to how to fix it, however it isn’t exactly the same as a 504 error or a 503 service unavailable error.
https://support.wix.com/en/article/live-site-error-error-502-bad-gateway

An important thing to remember is that the 504 error is mostly always a server-side error. That means the problem exists with the website you’re trying to access, and not with your computer. That’s both good and bad news. It’s good news because there’s nothing wrong with your computer, and it’s bad news because there’s usually nothing you can do to solve the problem from your end.

However, before you go complaining to Wix or others just remember that it can also be a client-side issue, or even a result of a third-party plugin, so it might still be something wrong with your code.

Wix uses a CDN (Content Delivery Network) server, so it could be a glitch somewhere with in their system that is causing your issue. however you are not to know until it gets found or sorted etc.

One of the easiest and first things you should try when encountering a 504 bad gateway error is to simply wait a minute or so and reload the page (F5 or Ctrl + F5). It could be that the host or server is simply overloaded and the site will come right back. While you’re waiting, you could also quickly try a different browser to rule that out as an issue.

Another thing you can do is to paste the website into downforeveryoneorjustme.com . This website will tell you if the site is down or if it’s a problem on your side. A tool like this checks the HTTP status code that is returned from the server. If it’s anything other than a 200 “Everything is OK” then it will return a down indication.

Whenever you launch your browser and visit a website it sends a request back to the web server it is hosted on. The web server then processes the request and sends back the requested resources along with what they call an HTTP header. This HTTP header contains one of many HTTP status codes to communicate whether everything is OK or if something has gone wrong. Not all HTTP status codes are bad. For example, a 200 status code means “Everything is OK.”

There are many different types of 500 status error codes (500, 501, 502, 503 504, etc) which all have different meanings. These indicate that the request was accepted, but the server prevented the fulfillment of the request.

In this case, a 504 gateway timeout error means that “the server, acting as a gateway, timed out waiting for another server to respond.” The code returns when there are two servers involved in processing a request, and the first server times out waiting for the second server (upstream server), to respond.

A 504 error can have a negative impact on SEO if not fixed right away. If your site is only down for say 10 minutes and it’s being crawled consistently a lot of times the crawler will simply get the page delivered from cache. Or Google might not even have a chance to re-crawl it before it’s back up. In this scenario, you’re completely fine.

However, if the site is down for an extended period of time, say 6+ hours, then Google might see the 504 error as a site level issue that needs to be addressed. This could impact your rankings . If you’re worried about repeat 504 errors you should figure out why they are happening to begin with.

Have a read of this page as it gives a clear description about it.
https://www.lifewire.com/504-gateway-timeout-error-explained-2622941
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504

Finally, are you using any HTTP functions on your website, as if you are then double check that they are setup correctly etc.
https://support.wix.com/en/article/corvid-exposing-a-site-api-with-http-functions
https://www.wix.com/corvid/reference/wix-http-functions.html
https://support.wix.com/en/article/corvid-accessing-third-party-services-with-the-fetch-api
https://www.wix.com/corvid/reference/wix-fetch.html

Ciao @givemeawhisky thanks for your large answer. I carefully read it, but I’m not able to test as you suggest.
I tried my routine with list of 10 only elements and I have to say that it works.

So I believe that, if it’s not the code, but the Wix database servers are the problem. They are really really slow .
There is another way of coding in order to skip this problem ?

Sorry to be a bearer of bad news, however if it is server side, then you can’t do much and it needs Wix to look into it for you.

So, hopefully Wix Mods see your post and act on it, otherwise I would suggest putting in a ticket to Wix Support so that they can pass it on to the relevant department.

They might just come back to you with the usual reply of this is a code issue, you are best going to use the Wix Corvid Forum, however if you don’t try you won’t find out.

Cross my fingers that somebody picks it up here or that Wix Support can actually pass it on to the relevant department and it gets sorted for you.:crossed_fingers:

Really thanks so much :slight_smile:

Hi @shiran-kohai how are you? One time you helped me, I’m really grateful, could you suggest now something about this strange behavior ? The problem is in the Wix virtual server and dedicated band width we have, Or something else?
Thanks in advance
Mauro

Guys, it´s very late here and I am looking at this code and I am trying to wrap my head around what is happening to the stack on the server when you first declare an async query and .then return the result as a promise. Couldn`t this be the problem, that this combination creates its own deadlock?

Hi @giri-zano I’m really grateful that you picked up my request of help!
But yet if I load my 1000 records .csv from file manager it works, so actually it must be something of wrong in my code, I hope. I’m looking forward to your news, thanks in advance
Mauro

Hi!

Well I see plenty of valid stuff that can be done in order to make it work.

The code itself is very heavy and can be refactored in a much more sufficient way.
As far as I understand - you are running a ‘for’ loop for 1000 times, and on each iteration you are triggering 2 DB calls while doing so by using ‘await’ per action.
This is a VERY heavy usage. Not for Corvid\Wix-Code… for any system…

My recommendations (per my understanding of your code) are as follows -

  1. Extract the data from “Soci” DB to an array instead of running a DB call per ‘for’ iteration so the compression action would be much lighter.
  2. Maintain a side array with all the diffs between the “Soci” DB (now represented as an array per step 1) and your “arraySoci”.
  3. Once you have an array with all the diffs - you can run a single bulkInsert() and a single bulkUpdate() command to update the “Soci” DB with all the changes (nice tip - you can do it faster with bulkSave( ) that actually do both insert and update in one command).

By doing this change you are basically converting ~2000 DB calls into 2\1 DB calls. It should be much lighter, and I believe it will do the trick.

In addition - in case that this operation doesn’t need to run explicitly by you, and can be run automatically once in a while - I think that you should check out our new job scheduler feature that can trigger backend functions on recurrent times.

Regarding the error that you’re getting -

When using a web-method (meaning a function within the backend that was triggered by a call from the frontend) we are running a process that is limited due to security reasons.
That time window is relevant when triggering an async function that retrieves a promise to the page (meaning the FE is ‘waiting’ for an answer, and afterwards kills the process).

If you want a process that lasts longer - you can call that web-method and from there trigger a different async function (and you should do that without using ‘await’).

@shiran-kohai reallyyy thanks. I had a doubt and I was thinking about a lighter solution. Sorry for my rough code :slight_smile:
I’m going to try your solution, I’ll give a feedback as soon
thx so much
mauro :slight_smile:

Hi @shiran-kohai I applied your advices. Wooow It’s a splinter !
I post the code for other people.
Thanks Sooo much !!
A lemon beer for you when You’ll be in Sorrento :slight_smile:
best greets
Mauro

import wixData from 'wix-data';

export async function aggiornaSoci() { 

// create Array 'Soci'
let options = {suppressAuth: true, suppressHooks: true}

let Soci =[];
await wixData.query("Soci", options)
.limit(1000)
.find()
.then((results) => {Soci = results.items})
.catch(err => {return err})


// create Array 'importaSoci'
let importaSoci =[];
await wixData.query("importSoci", options)
.limit(1000)
.find()
.then((results) => {importaSoci = results.items})
.catch(err => {return err})

//************************************************************************ */

// create array
let refreshArray = [];

// cicla l'array importaSoci
for (let i=0; i<importaSoci.length;i++){
let cn=false;
 
 // cicla l'array Soci
 for (let j=0;j<Soci.length;j++){
 if (importaSoci[i].tessera === Soci[j].tessera){
             cn=true;
             importaSoci[i]._id = Soci[j]._id;
             refreshArray.push(importaSoci[i]) }
            }
 if (cn!==true) {refreshArray.push(importaSoci[i]) }
}

console.log('refreshArray',refreshArray);


//************************************************************************ */
// REFRESH SOCI DATABASE
wixData.bulkSave("Soci", refreshArray, options)
.then(() => {return erase("importSoci") })
.then( res => {console.log(res) } )
.catch(err => {return err})


return "UPDATED DATABASE"
}


*/_____________________________________________________________________

export async function erase(dbase) {
let options = { "suppressAuth": true, "suppressHooks": true };


return await wixData.query(dbase,options)
    .limit(1000)
    .find()
    .then((result) => {
 for (var j = 0; j < result.items.length; j++){
            wixData.remove(dbase, result.items[j]._id, options);
        }
 return "Cancellazione importSoci OK !";
        })
    .catch((err) => {return err});

}

Did you ever get this fixed? Given the date, I assume so.

Did what get fixed? What are you trying to do? What code do you have? What works, and what doesn’t?

Please add your own issue into a new post instead of bumping up an old post.
Also, add any code in a code block as stated in the Forum Guidelines.

This is an old post and is being closed.