multiple insert to collection fails half way through!

[jay65866]

Not holding much hope of an answer due to previous experience on here but here goes:

I require many rows to be added to a collection at once, an average of 1500 a time.

If i was using a proper platform i would generate a large object and then insert all rows with one query.

so instead I loop the number of rows i need in a for each and insert with an await wix data insert. This is working up to around 500 or so rows where the system just stops reporting and nothing seems to be happening! checking the collection shows max 650 rows out of 1500!

does anyone please have any Ideas?

I do not have the time to write, yet another, function that checks how many and where the function failed and then re attempt to insert the failed rows. This will be extremely difficult as the data used to generate these rows could potentially be gone…

This is absolutely unreliable and unacceptable!

Thanks in advance

EDIT: I’m also using the script on this page: https://www.wix.com/code/home/forum/community-discussion/clear-collection-data-at-once
to remove all rows in a collection. There’s around 650 rows yet it keeps failing at 2-300 rows everytime! this is unacceptable!

[avillarg]

I have the same problem!

[J.D]

Use bulkInsert:
https://www.wix.com/corvid/reference/wix-data.html#bulkInsert

[avillarg]

@jonatandor35 Yes, I saw the link, but I don’t know how to use it when I’m working with many records (>300)

[J.D]

@avillarg using it is not more difficult than looping though the records and inserting them one by one, and it works better.
You can try writing the code, and if it doesn’t work, post your code here and get feedback.

[avillarg]

@jonatandor35 So I use the bulkInsert one by one record? In the same way as the Insert? I thought I could use it withe many records at the same time

[J.D]

@avillarg No. You put each record in an object, you push each object to an array and then you run bulkInsert on the array.
I may be able to help more if you add details regarding the data you have and its structure and how did you planned to insert it. But as of now your questions are pretty general…

[avillarg]

@jonatandor35 This is the code I’m using now (With Insert) How I convert to use bulkInsert?:

export async function RefreshInventario(event)
{
//Insert data from Rest Api into Collection
let idClienteSel = $w(‘#inputIdCliente’).value;

getInventarioSel(idClienteSel) 
.then( AllInfo => 
{ 

const cantRegistros = AllInfo.lenght;

//for (var i = 0; i < 50; i++)
for ( var i = 0; i < AllInfo.length; i++)
{
let toInsert =
{
“idCliente”: i,
//“idCliente”: AllInfo[i].IdCliente,
“nombreCliente”: AllInfo[i].NombreCliente,
“entradaNo”: AllInfo[i].EntradaNo,
“mp_esp”: AllInfo[i].MP_Esp
};

//wixData.bulkInsert(“colInventario”, toInsert)
wixData.insert(“colInventario”, toInsert)
.then(results =>
{
let item = results;
})
. catch (err =>
{
let errorMsg = err;
})
}

}) 

}

It uses this backend function:

export async function getInventarioSel(idClienteTxt)
{
let idCliente = parseInt(idClienteTxt,10)
//https://restclaripack.azurewebsites.net/api/inventario?IdCliente=198
const url = ‘https://restclaripack.azurewebsites.net/api/inventario?IdCliente=’ + idCliente;
console.log("Url: " + url);

try
{
console.log(“fun getInventarioSel”);
let response = await fetch (url, {method: ‘get’});
let data = await response.json();
return data;
}
catch(err)
{
console.log(err)
}
}

[J.D]

Try:

import wixData from 'wix-data';
 //.......your code........
//before the loop - declare an array:
let toInsertArray = [];
for (var i = 0; i < AllInfo.length; i++){ 
  let  toInsert =  { 
   "idCliente":     i, 
   "nombreCliente":  AllInfo[i].NombreCliente,
  "entradaNo":  AllInfo[i].EntradaNo,
  "mp_esp":  AllInfo[i].MP_Esp
 }; 
 toInsertArray.push(Object.assign({}, toInsert));
}//end of loop
wixData.bulkInsert("myCollection",toInsertArray)
  .then( (results) => {
 console.log("done");
  } )
  .catch( (err) => {
    let errorMsg = err;
  } );

[avillarg]

Great!!! It worked! Thank you!

[J.D]

@avillarg You’re welcome.

[giri-zano]

J.D., 2 questions:

1. why the

toInsertArray.push(Object.assign({}, toInsert));

and not

toInsertArray.push(toInsert);

(or, shorter, what does Object.assign add? Looked at MDN, but it´s not clear to me).

2. need to do something similar. Is there a memory limit to how big the array can be before you bulkInsert it?

[J.D]

@giri-zano Actually I’m not sure you have to create a real copy in this case. Maybe you can use

toInsertArray.push(toInsert);

try and let me know.
As for 2, I don’t know how big it can be, I guess you should test and see.

[doughammack]

I am lost at the field “idCliente”, is this a custom field or an autogenerated field? I am sorry if this sounds simple but is this the same as “owner”?

[giri-zano]

No, it´s got nothing to do with it. The “idCliente” is just a field name (also called “key” in Wix Data Manager) in a collection called (generically) myCollection. So if you have a collection where the client ID is called “clientId”, then you should use “clientId” in this array of objects before you bulk-insert them to the collection.

[arturovillarg]

I have used several times the wixData . bulkInsert without problems, but now, when I try to import about 3000 registers it doesn’t work, There is a limit to use it?