Performance and reliability issues with wixDataQueryResult.next()

Ask the community
, ,
1

Background
I have the following two codes. One that fetches a query without a limit and then uses native js slice to load slices of the data at a time. One that fetches a query with a limit and then uses Corvid code to get the next slice of data.

Using js slice:

async function loadProducts() {
 let productListAll = await wixData.query('Stores/Products')
 // .limit(productsToLoad)
    .find();
  productListAll = productListAll.items
  console.log(productListAll)
 if (productListAll.length > 0){
 if(productsLoaded === 0){
 let productList = productListAll.slice(productsLoaded, productsLoaded + productsToLoad)
      productsLoaded += productsToLoad
      showProducts(productList)
    }
 while(productsLoaded <= productListAll.length){
 let productList = productListAll.slice(productsLoaded, productsLoaded + productsToLoad)
      productsLoaded += productsToLoad
      showMoreProducts(productList)
    }
  }
}

Using Corvid hasNext() and next():

async function loadProductsViaNext() {
 let productList = await wixData.query('Stores/Products')
    .limit(productsToLoad)
    .find();
 if (productList.items.length > 0){
 if(productsLoaded === 0){
      productsLoaded += productsToLoad
      showProducts(productList.items)
    }
 while(productList.hasNext()){
      productList = await productList.next()
      productsLoaded += productsToLoad
      showMoreProducts(productList.items)
    }
  }
}

Issues (performance and reliability)
Right now I only have about 40 test products and the native JS code works fine and even seems faster. When I timed it using the date method (Corvid can’t use performance.now() for some reason) the JS method returned an average time of about 1500 milliseconds while the Corvid method returned an average time of 5000+ milliseconds.

Using Corvid’s hasNext() and next() also gives errors like:

Wix code SDK Warning: The data that was passed to data contained at least two items with the same ID: 8c90d74c-68aa-44bc-8140-0cb1de0ed30b. Only the first item was accepted.

However, it is inconsistent in that it’s not always the same ID in that error. It also causes items to not load properly as it loads the wrong ID. In theory

productList = await productList.next()
productList = productList.items

should return the same thing as

 let productList = productListAll.slice(productsLoaded, productsLoaded + productsToLoad)

but .next() returns inconsistently incorrect values. Sometimes it’s only 3 Id errors, sometimes it’s over a dozen. The only consistency I could find is that it seems to happen to items in the middle no matter how the items are sorted.

Questions
Why is this happening? How do I fix this? Why is Corvid code involving query so slow? Is it even worth figuring out how to fix this issue or should I just set it up to use as much native JS as possible to improve performance?

2

I myself exclusively use the second (next) method, either with queries or using a dataset, and never have any trouble.

From what I understand, showMoreProducts(productList.items) adds new additional items to a repeater. What does that function look like? The error message that you are getting refers to the data property of the repeater.

Also, what does the productsLoaded variable look like using console.log() ?

3

The productsLoaded variable is just the number of products loaded. It was mainly used in the JS code using .slice() and to detect whether showProducts() needs to be run . Yes you are right showMoreProducts() adds new items:

function showProducts(loadedSet){
  $w('#productsRepeater').onItemReady(productItemReady)
  $w('#productsRepeater').data = loadedSet
  $w('#productsRepeater').show('fade', defaultFadeOptions)
}

function showMoreProducts(loadedSet){
 let data = $w('#productsRepeater').data
  $w('#productsRepeater').data = data.concat(loadedSet)
}

I initially didn’t have a showMoreProducts() but I found that the .show() would run repeatedly for the repeater instead of each repeated element so it’d basically keep flashing.

I have changed the code to load the productListAll in another function with the following since I found that I needed the productListAll to determine filters so might as well make it a variable I can use again without query again.

async function getProductsList() {
 let productList = await wixData.query('Stores/Products')
    .limit(100)
    .find()
 let productListAll = []
  productListAll = productListAll.concat(productList.items)
 while(productList.hasNext()){
    productList = await productList.next()
    productListAll = productListAll.concat(productList.items)
  }
 return productListAll
}

then

let productListAll = await getProductsList()
loadProducts(productListAll)

which seems to work fine and seems just as fast as my JS code. However, for the sake of learning I’d still like to know what went wrong in the initial code and would appreciate any insight on the matter.