How to prevent or delete duplicate records in CSV file import process to wix database? Code example

Ask the community
,
1

Hi;
I add csv files to my wix database, since the number of records is high, I cannot delete or compare them one by one.

I transfer the data in the csv file that I update everyday. But I do not want to reload the data I transferred the previous day. Each record must be unique.

Do you have a sample code? such as automatic deletion of duplicate records or ignoring file upload

I’m trying the two codes below but I couldn’t. Where’s my fault? How should I do a sample code or educational video about it?

I want to prevent repeated records in the database. It is not possible to manually check or delete the CSV file every day, since there will be more records in the file, which will contain repetitive records every day. Please help with this.
Thank you

Note: The column where I check the duplicate records is the original link url address / column with url content.

https://stackoverflow.com/questions/62465609/how-to-prevent-or-delete-duplicate-records-in-csv-file-import-process-to-wix-dat

reference column for comparison : OriginalConnectionDetails
(By comparing url addresses, I do not want to add the existing record back to the database while transferring the csv file if it was added to the database before.)
Database : Germany

Example Code-1

import wixData from ‘wix-data’; export function searchForDuplicates(value) { return wixData.query(“Diamond”) .eq(“giaCode”, value) .find() .then((results) => { return results.items.length; }) .catch((err) => { let errorMsg = err; }); } export function Diamond_beforeInsert (item) { return searchForDuplicates(item.giaCode).then((res) => { if(res > 0) { return Promise .reject(“Duplicate”); } return item; }); }

-------- Example Code-2

import wixData from ‘wix-data’ ;

export function searchForDuplicates(collection, field, item) {
// info contains the hook context
// use the collectionName property to use function for multiple collections
return wixData.query(collection)
.eq(field, item[field])
.find()
.then((results) => {
return results.items.length;
})
. catch ((err) => {
let errorMsg = err;
});
}

export function Germany_beforeInsert(item, context) {
// Calls routine to check if value already exists in the collection.
// If value not found, then save record.
// Else, if value is found, then reject this insert to prevent duplicate values.
// Note: concurrent inserts may result in duplicates.
// Pass context to searchForDuplicates to use for multiple collections.
return searchForDuplicates(context.collectionName, “OriginalConnectionDetails” , item).then((res) => {
if (res > 0 ) {
return Promise.reject( ‘This item already exists’ );
}
return item;
});
}

#database duplicate #csv #csvimport #databaserecorddelete #delete #duplicatedelete

2

Hi Umut :raised_hand_with_fingers_splayed:

Before uploading the CSV file, make sure that each entry has its ID ( _id ) and the ’ _owner ’ fields included, and sync them when you import the CSV file (don’t ignore them).

And, as an extra tip, you can easily edit your database from your dashboard instead of exporting and importing the database over and over, this will prevent a lot of hassle and the wasted time.

Ahmad

3

Hi; Ahmad

I added the code to the hooks. But it didn’t work.

I didn’t understand why it didn’t work or where I made a mistake.

I am trying to do; Importing the data file with the same number of columns and the same format, everyday contains approximately 1000 records. But there are only 50 or 100 new records in 1000 records.

By comparing the original link url address, if data has been previously transferred to the wix database (eg germany) (if there is data with the same url address), to prevent the occurrence of repeating original link url information, to avoid renewed registration.

When importing the csv file I want to ask (in the same column information) can I compare the original link address column and check the existing data to add only new information?

Or am I in the wrong way with the code ?

748×352 33.9 KB

1030×588 43.1 KB

import wixData from 'wix-data';

export function searchForDuplicates(value) {
 return wixData.query("Germany")
    .eq("OriginalConnectionDetails", value.OriginalConnectionDetails)
    .find()
    .then((results) => {
 return results.items.length;
    })
    .catch((err) => {
 let errorMsg = err;
    });
}

export function Germany_beforeInsert(item, context) {
 return searchForDuplicates(context.collectionName, "OriginalConnectionDetails").then((res) => {
 if(res > 0) {   
 return Promise.reject("Duplicate");
    }
 return item;
  });
}
4

If anyone has found a solution in this matter, can you send me a solution?

No one other than me has sought a solution in this regard? …