I have some code that creates a set of data for each user. They can regenerate this at anytime they like. But before the code runs that generates new data, first I clean up the old data and to do this I use bulkRemove. To add I use bulkInsert.
async function deleteMeals(){
let removeList = []
let myMeals = await wixData.query("myMealsTest")
.eq("_owner",wixUsers.currentUser.id)
.limit(200)
.find()
for (let i = 0; i<myMeals.items.length;i++){
removeList.push(myMeals.items[i]._id)
}
console.log("removing",removeList)
await wixData.bulkRemove("myMealsTest",removeList)
}
async function startPlan(){
await deleteMeals()
var mealPlan = []
await calculateMealsNew(mealPlan)
//A bunch more stuff
}
I would say this runs about 300-400 times a day. And about 1% of the time it just doesn’t work. There should be between 35-42 entries for any 1 user maximum. These should all be deleted before the new ones are added (calculateMealsNew takes about 250ms to run before it starts adding meals). I have never been able to replicate this. For me the delete always happens before the calculation even begins.
These are triggered by user action. Usually by a few clicks and inputs so the page would always be ready by the time these trigger. I upped the limit to 200 trying to fix this for people who had multiple data sets stacking and it couldn’t delete all of them at once.
It seems to effect someone multiple times more frequently than it should. There are people with 100+ entries meaning it has failed to delete 2+ times and these failures occur on different days. I check the created dates of the entries and many times they are hours or days apart. But for the same person it just doesn’t delete somehow more than once.
Sometimes it partially deletes data. I had situations where it deleted about half of the data but not the other half. It deleted about 20 records but left 20. This happens fairly often and is even more bizarre to me.
Is there some better way to delete all of a users old data before creating new? Or some way to make this more reliable? It works 99% of the time but I get 3-4 people every day that this happens to and it requires me to manually go in and fix it.
First, I would add catches to log your errors. You’ll have a better time auditing your code edge cases.
Secondly, do you execute that function on the backend of frontend? The backend has a limit on how long a request can run. So my suggestion would be to do your operations in 2 times. First delete, then Insert. That way you have 2*14 second for your script to execute.
You can also include a retry mechanism in your call. If the error is a timeout, simple recall the procedure (limit yourself to 1 or 2 retries otherwise Wix will block your calls for 1 minute)
Lastly, you could change a bit how the system works: Add a new field “version” to your collection. When you insert a new batch get the latest version and increment it. When you need to retrieve the items add a filter on the version field so you only get the items from the last batch. This way you never delete anything, you only insert. Bonus point, you get a new history feature where you can get history of each user
It is done on the frontend, typically it takes a very short period of time under 1 second total for delete and for the insert to finish. Version I have thought of but the amount of data would get very large potentially 1 million + records is this ok?
Also I know about catching errors when testing but I have never had this happen for myself or any of my testers. There is far too much activity for me to use the monitor and require me to sit there for 4 or 5 hours waiting for it to happen hoping it just doesn’t scroll by before I can read anything. Maybe some type of permanent log?
In addition to what @plomteuxquentin wrote, you might want to try LOWERING the limit instead of upping it (set to 200 now). wix-data .bulk-functions are still a bit of an enigma. The documented limit is 1000, but Content Manager deletes a 1000 rows in chunks of 20 to 40. I would try to set a chunk size, like 40, calculate how many to delete, how many chunks that is, etc, then delete bit-by-bit.
Also, as @plomteuxquentin said, you must add error catching, otherwise you will have no idea if the underlying problem might be a (recoverable) WDE00053 or a fatal one, if any.
Does this make any sense?
I personally have a mechanism to store all the errors in a DB along with the page, function and location of the error origins, and return a support ID to frontend for the client to use when they contact us, the point is, I always know when and what errors occur, and with whom.
Try to lower the chunks of data to 50s or so, add a retry mechanism when a timeout accur, and always, always log or store the errors when they occur, the more data you collect when one occurs the better.
async function deleteMeals(){
try{
let removeList = []
let myMeals = await wixData.query("myMealsTest")
.eq("_owner",wixUsers.currentUser.id)
.limit(200)
.find()
for (let i = 0; i<myMeals.items.length;i++){
removeList.push(myMeals.items[i]._id)
}
console.log("removing",removeList)
await wixData.bulkRemove("myMealsTest",removeList)
let options = {
"suppressAuth": true,
"suppressHooks": true}
wixData.insert("errorlog",{errorMsg:"successfulInsert"},options)
}
catch(err){
let options = {
"suppressAuth": true,
"suppressHooks": true}
console.log("error is",err)
wixData.insert("errorlog",{errorMsg:err},options)
}
}
So I tried this and so far no errors but still issue continues. For some reason even though I have permissions on this particular collection wide open it is only writing to it as an admin so I did the auth suppress and it does work. I added a write for now just in cases of success just so I know that at least SOMETHING should write. I will try reducing number of deletes at once (it should honestly never be above 50, only time it is more than that is if the error already took place)
Hey, you said you were using it on the front end but on the other hand your code includes suppressAuth which can only work on the backend.
Try to see if this is the problem.
@jonatandor35 I just recently added that. The insert doesn’t work at all unless an administrator does it. The collection is set to site content so not sure why it doesn’t work without it.
I am getting the success insert any time a user generates a new meal an but have yet to see any failures even though I am still having the delete problem occurring. Just no error happens
@archaeous site content means your data is viewable by anyone but only editable by the owner.
What you probably need is member-generated content or so custom permission where only the author can read and modify the data.
@ahmadnasriya & @archaeous I believe you are wrong trying to log the issue into a Wix database. What if the issue is with the WixData? You log will failed too. Have a look at google operation it really does a good job. Especially when combined with the technique of Ahmad that automatically generate a ticket
I totally agree with you, but before I store the error in a database, I always console.error() the actual error so it also appear in my Google Operations log as well, so even of wixData fails, I still have a copy of the log in Google Operations.
It seems to be generating a log fine from a technical perspective. I have many rows of success messages.
It doesn’t generate unless I have the suppressAuth on
There are no failure messages, but it is failing at times I am getting a success message
I checked the times of the log vs the times in the collection that is having the issue. There are entries that should have been deleted from the time where there is a success message.
I just recently added a field to the log that tracks the array that was passed into the bulkRemove. Maybe then I can see if it is passing an empty array or a partial one (I do still sometimes get half deletes)
One user had it fail 3 times in a row for them.
They had a successful dataset generated sometime mid June
They decided to regenerate today.
Their old data was not deleted and they saw double the number of records they should. But the log said it was successful
They tried 2 more times and each time new data was inserted but old data was left. Log entry was created each time with a success message.
This user is set up the same as about 800 other users (they purchased plans)
@plomteuxquentin@giri-zano
One more update I think I found the issue some of the time but I don’t know how this could possibly happen.
I perform a query to get the data based on the owner. This is one of the keys for most tables like this as it is unique for each user. This is used everywhere this table is used.
let myMeals = await wixData.query("myMealsTest")
.eq("_owner",wixUsers.currentUser.id)
.limit(200)
.find()
I noticed some of the queries are using a very strange owner id
This is a query that returned nothing understandably as that _owner doesn’t exist. However the entry created in the log is for the correct user. These lines of code are just a few apart so I don’t understand how this can happen. How can the wixUsers.currentUser.id retrun some very large string and yet a few lines later correctly use the id to create a row in the table
How do you make sure the user calling that function is logged in?
Have you set the permission of your module to logged in member only?
I don’t know what happening there, but maybe some try to call your function before Wix has authenticate the person (you are a site member, your session run out, you come back to the website your old session is refreshed) and that MIGHT be the reason why this is happening (again this is a wild guess)
In order to view the pages the user must be logged in and have purchased a plan. In addition when it creates the data, and logs it, it is using the correct Id (This is the automatic creation of the owner when an insert happens though. I am not manually assigning the Id)
So the automatic writing of the owner happens just fine without fail. But sometimes wixUsers.currentUser.id is returning something very strange. I have a feeling this is causing another one of my issues too where it is creating duplicate main user records as it checks to see if they have a record, if they don’t then it creates them an initial row in a member data collection.
I am surprised the pages don’t detect they have been logged out because in order to do this a user must click a few times and input a couple of fields so I would think if they were logged out they would be asked to log in again. Especially on a restricted page. But that kind of makes sense. That ID doesn’t match anything I see and could be like a unique session id. I am going to see if I can see what this ID looks like for a user that isn’t logged in somehow
I’m with @plomteuxquentin here. That long ownerId is a 100% proof of a situation where a non-loggedin user accesses that page, no doubt about it. Don’t you have a code-leak somewhere, I mean, some function that (accidentally) lets non-logged in users access that function?
@giri-zano I dont know how, the entire page is restricted to only logged in paid members. Even regular members don’t have access.
The member in question is a paid member. The crazy thing is they at the same time have other data being written and it correctly uses their right id. I mean even in the error log itself the owner of the row is the correct owner, but in the query it doesn’t have their ID. So I would expect that if they weren’t logged in there would be something wrong with the owner of the row, but that works without
` in any case could you try switching the permission on for the javascript web module? If this is the issue it will prevent unlogged user to call the backend function (see settings on the .jsw)
Let us know if this help otherwise I would contact support explaining them you have a problem with currentUser.id returning strange value
I’m confused. Where is your code? Frontend (page) or Backend (web module)? I see in your code that you appear to be using wix-users (which is a frontend API), and suppressAuth (which only works in the backend).
If your code is in the backend, you should be using wix-users-backend , as wix-users won’t work in backend code.
If your code is in the frontend, the suppressAuth won’t work.
Also, keep in mind that wix-users only has partial functionality in Preview mode.
Database queries should always be in the backend to protect sensitive information and protect your data and your users’ data.
