I’m pulling about 1 Gb of data from an external database via fetch calls. I iterate through the paginated data, and bulkSave it to my Wix collection for analyzing.
For obvious reasons, a recursive function pulling that amount of data times out on the backend, but I discovered Job Scheduler doesn’t share the same timeout limitations.
Unfortunately, I’m noticing the job gets killed after 7 minutes of running. Thoughts? Solutions? Thank you!
Actually, the backend has the same limitations as Job Scheduler. I guess your first experience with a timeout was 14 secs. That’s a frontend timeout (waiting 14 secs for a backend call to return something). Because Cron Job have no frontend, that limitation does not exist, so you got the full container spin cycle, which is five minutes plus 60 secs grace period. Then the container times out and everything stops, no way to recover.
So for now, I cannot give you any quick solution, except this one: set a backend timer and after 5 minutes, start to close gracefully, set a marker somewhere (e.g. in a separate collection) where you left off and let a second Cron job take it from there, etc.
Unfortunately, the minimum interval, currently, for Job Scheduler is 1 hour, so it will be a long night Job.
Wish I had a better answer, but it’s the best I can do for now. I know something is in the works at Wix, but no ETA, so you will have to work with what’s available now.
P.S. Have you checked your bulkInsert’s? The max is 1000 at a time on paper, but nobody achieves that, we all do it in chunks of like 50 or a 100 max.
Thank you so much for the thorough and quick response! The 5 minutes + 60 second grace period is a helpful data point!
The 14 second timeout was triggering when I ran the “test function” - little play button in the Velo editor - despite the function being declared in the backend.
Fortunately, this big bulk import only needs to happen once, then I can run an hourly/daily cron job that pulls newly updated records.
API call delivers JSON in 100 record pages, so BulkSave has been perfect. I was considering saving a marker and chaining together 6 cron jobs that pull that marker and update it based on the api “&$skip=” header. Thank you for sharing the suggestion! It helps validate a solution for this initial bulk import without me babysitting and republishing the jobs.config every 7ish minutes. Haha.
I appreciate your insight!