Scaling image generation: queues, idempotency and retries

Image generation breaks the assumptions most web architectures are built on. Requests take dozens of seconds rather than milliseconds, capacity is genuinely finite because it is physical, and a failed request has already cost real money. The patterns that keep this correct are well known — they are simply not the ones a typical CRUD backend already has.

The queue is the design

The first instinct is to call the API directly from the request handler. It works in development and fails the first time real traffic arrives, because there is nothing between demand and capacity: a spike becomes a wall of timeouts, and there is no record of what was in flight when the process restarted.

Put a durable queue in the middle. "Durable" is the operative word — an in-memory list loses every in-flight job on deploy, and deploys happen during traffic. A database table works perfectly well at this scale and gives you a recoverable record of state as a side effect.

Backpressure is a feature

With a queue in place you can decide how many jobs are allowed in flight at once, and that limit is a product decision rather than a technical detail. Too high and everything is slow for everyone. Too low and capacity sits idle while a queue builds. What matters is that the limit exists and is enforced in one place, so that a traffic spike lengthens the queue instead of degrading every request.

Idempotency, and the ambiguous failure

The failure mode that costs money is not an error response — errors are easy. It is the ambiguous case: the request left your process, the connection dropped, and you cannot tell whether the job was created. Retry it and you may have paid twice. Do not retry it and a user may have paid for nothing.

An idempotency key resolves this by making the retry safe: the same key returns the original job instead of creating a second one. The rules are simple and easy to get wrong — generate the key when the user acts, not when the HTTP call is made; store it with the job record; reuse it for every retry of that action.

# same key for every retry of one user action
KEY=$(uuidgen)
curl -X POST https://api.example/api/v1/jobs \
  -H "X-API-Key: $API_KEY" -H "Idempotency-Key: $KEY" \
  -F image=@photo.jpg -F feature=undress -F consent=confirmed

What not to retry automatically

There is a category of failure where the safe move is to stop. If your process died between sending a submission and recording the response, you genuinely do not know what happened — and without a key to deduplicate against, an automatic retry is a coin flip that can bill twice. Mark those jobs as needing attention rather than retrying them blindly. A visible stuck job is a better outcome than a silent duplicate charge.

Reconciliation: the loop that saves you

Live events are convenient and unreliable. Sockets drop, processes restart, and the one message that mattered arrives while nobody is listening. The fix is a background loop that periodically asks the API about every job your database still considers active, and updates state from the answer.

This single loop absorbs an entire class of bugs. Missed events stop mattering. A restart mid-generation stops mattering. Every job converges to its true state within one reconciliation interval, whatever happened to the connection in between.

Progress without hammering your database

Generation emits progress frequently — potentially several times a second per job. Writing each tick to your database multiplies your write load for no additional information. Throttle persistence to something like once per second and always write the final state, then broadcast the live values to connected clients without touching storage at all.

A checklist that holds up

• Durable queue between users and the API, recoverable after restart.
• An explicit in-flight limit, enforced in one place.
• Idempotency key generated per user action, stored, and reused on retry.
• Ambiguous dispatch failures flagged rather than auto-retried.
• A reconciliation loop that converges state regardless of missed events.
• Throttled progress writes, live progress over the socket.

None of this is specific to image generation — it is ordinary distributed-systems hygiene. It simply becomes visible here, because each unit of work is slow enough and expensive enough that the usual shortcuts stop being invisible.

Frequently asked