Scheduled Job with 59 second timeout

Ask the community
, , ,
1

I’m having trouble with
running a backend scheduled job through jobs.config which times out after 59 seconds because it hasn’t finished (which I believe is the limit for a backend job even on a Premium plan).

Working in
Wix Studio - I don’t need to share the code as its not a code issue….

What I’m trying to do
I have a lot of experience in Velo so, bear with me this is more troublesome than it seems.

I have a client site (Premium) with a CMS table with 116 rows (varies by day). These are properties. Each property has a media gallery that has between 1 and 25 images in a media gallery. So potentially a lot of images…

I need to run a scheduled job overnight that generates XML data for each property. This is for a 3rd party site like Zoopla. Don’t worry about the format etc that’s not the issue here. Ordinarily ,1 single scheduled job could trawl through the 116 rows and spit out the XML (function included in the scheduled job). Would probably take 30 seconds at most - so well within the 59 second limit.

However as WiX holds the image URLs as internal wix: format URLs, I need to make a backend call ( from the backend scheduled job) to the media library getDownloadUrl() function. No problem. Works just fine.

However, as that call is not the fastest - I only get to process about 7-10 of the 116 rows before 59 second have elapsed and the scheduled job dies !?!

Because 59 seconds is a limit I have read about for a backend function on multiple threads. And of course scheduled jobs run in the backend so is killed after 59 seconds.

Simples I thought. I’ll schedule the job to run every minute for 30 minutes…. but each time it would be able to pick up from where the last job stopped (knowing how many rows have been successfully output to XML so far….). No problem in testing, works a treat when running from the front end. But from the backend - it seems you cannot repeat a scheduled job unless it’s 1 hour after the last ???

So I now have a simple job running on the hour from 8pm to 6am to get 59 seconds worth of grunt with 11 attempts over 11 hours.

Has anybody else had this issue with scheduled jobs where you have a data function that takes more than 59 seconds?

Is 59 seconds per backend job a thing for a Premium plan ?

Is a 1 hour gap between scheduled jobs a restriction people are also working around ? ( I know its in the documentation that its 1 hour)

Any workarounds anyone has tries ?

I thought of auto generating the downloadable URL for each image when I store the images in ythe first place (then never have to call the getDownloadURL at all from the backend scheduled job) but the downloadable URL has a time to live of 1 year - which would create a ticking time bomb for properties that don’t change in a year !. Also, it creates significantly more maintenance when adding, removing images etc in the front end

Any help appreciated. I do have a ticket in progress too.

thank you in advance.

1 Like
2

That’s interesting. Are you using Promise.all to speed up the call by any chance or are you treating each item atomically? When reading this, it does not sound like it should be an expensive computation on the server. I know Wix made changes to their server side payloads a while back which does impact the sheer size of the calls we can make but for what you’re trying to do, this doesn’t sound like an issue. Even 116 rows of data is not a large collection size. How large are the files in your galleries? Are they videos and images?

Also try and create your gallery field as an index in your CMS to speed up the query find operation if not already.

I do not know your call sequence but the getDownloadUrl only does one file at a time and that will surely result in a timeout. perhaps you’re able to use the downloadFiles API?

Or to your idea of storing a temporary URL, you can do that too. Sure it’ll expire in a year but you can just clone the gallery unique id item with the file URL in a new CMS that can act as a cache. This CMS can have timestamps into when the url will expire and have a job just get a new URL before it expires if it still exists. By doing this, your server will be quicker since it works atomically and can handle more bandwidth.

1 Like
3

Hi, thanks for your response, appreciated.

In answer to your questions / clarifications.

  1. Promise or await - either is the same - its not so much a speed issue more that the getDownloadURL is slow enough irrespective of the synchronous / asynchronous method of function call (IMHO). Each getDownLoadURL can be several tenths of a second or maybe a second in some instances ( nothing to do with the size of the image - they are all the same and just images not videos). Its just the resources of the backend servers .
  2. You’re right 116 rows is nothing. However on average each property will have 12 images. Its the downloadURL calls for each that’s causing the function to hit 59 seconds before completing even 15 rows (with 12 images avg each row).
  3. Its not the indexes, the 116 rows are already loaded into memory at the start of the function - within 3 seconds of the function starting…. it’s the calls to get the URL that takes the time….
  4. the downloadFiles API provides a “page” that allows direct access to the downloaded images…. its not a list of URLs - which can be used in an XML feed.
  5. Not worried about how to store the downloadable URL when the images are created - I can code that - just seems a lot of extra uneccessary work to workaround a WiX restriction on a 59 second backend job when one of it’s APIs (getDownloadUrl) is so slow. Makes it impossible ! ….. unless someone has figured a way ! …. thank you though.
4

Hi, @ptiernanhome !!

I haven’t done it myself, but there’s something I’ve always wondered about. In situations like this, wouldn’t it be possible to set up an external service—like Google Apps Script—to call the APIs you define on Wix at regular intervals? That way, you could run the process more frequently than Wix’s built-in scheduler allows. Of course, if you do this, you’d need to control the frequency so you don’t put unnecessary load on Wix’s servers. After all, that’s what the built-in limits are there for. :innocent:

2 Likes
5

Gotcha. Storing the downloadable URL as a cache would be a workaround to your issue, albeit a more time intensive task. But have you checked out the docs for wix media v2? They have new calls to get permanent URLs, compressed files, or individual URLs. This seems like a much better route.

You could essentially just map the fileUrls directly from the gallery and pass it to this endpoint. The endpoint will then provide you with the URLs you need as permanent links.

6

Thanks. Yes I did checkout v2… not sure they help but thank you.

I have changed my code to call all 1-25 download url calls per property row to run in parallel… so the overall time to generate 1 property xml is reduced. The benefit is I can process more property rows in the 59 seconds I have for the scheduled job time… therefore reducing the number of repeat cron/scheduler to 3 calls over 3 hours (having to be 1 hour apart) rather than 11 hours. Just managing the API quota per minute restrictions now..!!! Very painful with the low levels of throttling on backend call time (59 secs) and unpublished quotas (not easy to find for a premium plan) and I understand these are per minute & per month. So painful & so restrictive. I will publish any updates tomorrow following further tests & changes. It should not have to be this difficult to generate xml for 116 rows including images. Thanks for your suggestions. Appreciated.

7

Also think on how your CMS gallery gets updated. You can combine the permanent URLs with a new array field in the row and utilize event hooks on collection changes when its updated/created to map the URLs along with the gallery UUID. That way your scheduled job skips the time intensive task and the event hooks update the item atomically when needed. That is the idea of the cache I am referring to, that way your scheduled jobs only has to query and map your array fields into xml.

1 Like
8

Got it. Thanks. I’ll update in progress tmrw. Cheers.

9

Ok.. so I kicked off the getdownloadurl funtion in parallel for all images for a given property. I thought running in sequence would be kinder to the backend !! However it means I get through more properties before the backend job dies due to the restrictive timeout on backend jobs.

I got through 57 properties in 1 call to the backend function. Each property having the getdownloadurl function called in parallel for the 1-25 images in its gallery.

So now I will kick the job off overnight every hour for 4 hours. Each call picks up where the last left off.

So 2 calls (restricted by WiX to around 59 seconds each) likely to complete the task of 116 properties. Calls 3 & 4 likely to find nothing to do.

Will update tomorrow on progress.

Summary view.. having such a short timeout in a backend job that requires CMS processing is poor especially on a Premium plan. Also restricting scheduled jobs to 1 per hour is also very restrictive as the jobs can only run for 1 minute before they are killed. !!! But you cant run another for an hour :slight_smile:

1 Like
10

@onemoretime thanks for the suggestion. I may look into that but as of today it may work albeit with 3 or 4 calls to the backend job overnight !!

11

Scheduled jobs:

  • Free sites and most premium plans: You can add up to 20 scheduled jobs that run at a minimum of 1 hour intervals.
  • Elite and Business Elite premium plans: You can add up to 30 jobs that run as frequently as once every 5 minutes.

If you need them to run more frequently, you could schedule multiple jobs, 1 for on the hour, one for quarter past, one for half past etc.

You can use getFileDescriptors which allows you to pass an array of the Wix Media URLs in bulk (docs). For example:

["wix:image://v1/0abec0_b291a9349a0b4da59067f76287e386fb~mv2.jpg/leon.jpg#originWidth=3024&originHeight=4032"]

It’ll return an array where you can access the url, which is in the format of a WEB_URL

Screenshot 2025-12-08 at 15.00.40
Screenshot 2025-12-08 at 15.00.40681×111 4.37 KB

I imagine you can run these requests concurrently in batches too, along with bulk updating the CMS data on completion

12

thanks @noahlovell . Really appreciate the response. So it looks like getFileDescriptors() returns static URLs that can be used externally ? Do these have a time to live like the getDownLoadURL or permanent while the image is still in the media manager ?

…that would be great and I will switch to that for efficiency purposes rather than getDownLoadURL() which I have to call for each image (with average of 12 images per property row) - even though I call the downloadURL function concurrently for all images on a single property row.

Just a couple of other observations. I have a low end premium plan for my client. The scheduled job runs at 1am but dies after a short period (less than 5 mins). As I said before it just dies. No Errors. Also at a random position in the 116 rows being processed. Last night it died on row 75. I schedule the job then at 2, 3 & 4 am. The 2am job picked up where the last left off and processed 41 rows to completion. So the 3 & 4 am jobs had nothing to do. I can’t set the jobs to run at 15 mins & 30 mins etc past as that breaks the rule on my premium plan to being at least 1 hour between them. :slight_smile:

I suspect if getFileDescriptors works more efficiently I may get all 116 rows to get processed in one function call. Any ideas why these jobs just die ? and not run for 20 minutes if required ?

thank you.

13

Correct

Unless otherwise stated, these should be static

That’s on the level of an individual job, however you can setup multiple jobs (up to 20 on your plan) to run at different intervals, but call the same function. For example:

// Jobs Config Generator
// https://wix.to/NDAQn6c

{
  "jobs": [
    {
      "functionLocation": "/filename.js",
      "functionName": "funcName",
      "executionConfig": {
        "cronExpression": "0 * * * *"
      }
    },
    {
      "functionLocation": "/filename.js",
      "functionName": "funcName",
      "executionConfig": {
        "cronExpression": "15 * * * *"
      }
    },
    {
      "functionLocation": "/filename.js",
      "functionName": "funcName",
      "executionConfig": {
        "cronExpression": "30 * * * *"
      }
    },
    {
      "functionLocation": "/filename.js",
      "functionName": "funcName",
      "executionConfig": {
        "cronExpression": "45 * * * *"
      }
    }
  ]
}

It sounds like what you’re running into is a platform-level thing rather than an issue with your code. When backend code executes, it runs inside a container - and each container has a fixed lifecycle: it stays alive for up to 5 minutes, plus a 1-minute grace period for code that began running within the 5 mins.

Your job happens to be running inside one of these containers. Even if your function needs 20 minutes, the container will simply shut down when it reaches the end of its lifecycle, and your function dies with it.

There’s also a pool of warm containers that the system pulls from. Each incoming request gets assigned to whichever warm container is next in the pool - but you don’t control where that container is in its lifecycle. So your code might start in a container that only has 30 seconds left (before the grace period), or one with several minutes left. Either way, it will never exceed the ~6-minute max.

That’s why you’re seeing jobs stop “early”. The backend request timeout will vary according to plan

14

@noahlovell thanks Noah. I changed the job to use getFileDescriptors()… much quicker ! I needed to add a wait for 1 second per row to avoid a quota limit breach on updating the property row & inserting the XML row (2 data calls per property)… so hopefully overnight tonight it will work ok. I changed the scheduler to run the same job with 4 entries 15 minutes apart so hopefully that works too.. It seems the 59 second front end to back end timeout (to test the functions) isn’t absolutely clear in the documentation and I didn’t find anything saying that backend jobs (called from frontend or scheduler) will timeout (die) after 5-6 minutes !. However, the 2 workarounds above should solve the problem. Will update tomorrow. Again, thanks for your help

15

@noahlovell @oakstreetassociates Hi both. ok so definitely the getFileDescriptors() function is far more efficient on the backend than getDownloadUrl() and returns static URLs for the images. I had to install a WiX Media package for it to work. My scheduled job ran in less than 3 minutes and converted all 131 properties (new total !). Perfect. The 2nd, 3rd & 4th scheduled jobs (same job picking up where the last left off) - at 15 minute intervals (as suggested - not 1 hour apart) picked up 0 rows to do so weren’t needed - but there if in the future not all rows can be processed in one scheduled call.

Perfect. Thank you both for your assistance, direction and ultimately resolution. Appreciated.