Data retrieval from database, faster alternatives?

Ask the community
,
1

I need to query almost 300k items from a database in backend. The most straightforward way of retrieving data 1000-by-1000 (using skip) works, but is really slow. It’s no option to use the .hasNext of a query in back-end, since then the site can only be populated with the data after all data has been retrieved. (Takes at least a minute, and users of course do not want to wait that long). I have tried using the google sheets api as well as googleapis modules, but both don’t seem to work (anymore).

Question thus boils down to: are there any other ways of querying data from a database that are significantly faster than the way I am doing it right now*, using wix/corvid api/code? Does anyone know of other (node_)modules that could be helpful in this instance, with proven, or self-experienced executability?

Thank you for your input.

*As described earlier, I have tried 2 ways: querying in front-end 1000-by-1000 using skip in backend (database has restricted access), or use the query .hasNext method until all data has been retrieved. Both take way too long compared to modern-day UX standards.

2

Hi,
As an option, you can try splitting the data between multiple collections and run multiple queries simultaniously. Then combine the results.

3

Hi Aleks,

Thank you for the great alternative . I have implemented your suggestion and expected it to work quite well. However, it seems that Wix can only have so many outstanding promises running ‘simultaneously’. There seems to be a limit on either memory or cpu resources that can be used.

To anybody having the same issue now or in the future, here’s what I’m doing now :
I have json files stored in backend containing a (json) representation of the data I want to load. I then ‘require’ (see https://stackoverflow.com/a/7165572)) them as a json object 1-by-1 until all of them are loaded to front-end. Takes around 55 seconds for all of them to load. Slow, but makes the best of the worst.