Background Jobs in Production: The Problems Queues Don’t Solve

Moving work out of the request path is one of the most common ways to speed up backend systems. Emails are sent asynchronously. Invoices are generated by workers. Webhooks are delivered through queues. Image processing and indexing run in background jobs. Latency improves immediately. But many teams eventually notice something strange in production: duplicate emails appear retries increase system load dead-letter queues slowly grow workflows technically "succeed"... but the outcome is wrong The queue is healthy. Workers are running. Yet the system behaves incorrectly. Moving work to the background changes where failures happen . It does not remove them. This post is a shorter version of a deeper engineering write-up originally published on CodeNotes. The Assumption Behind Background Jobs Background job systems are usually introduced with a simple expectation: If a job fails, the queue will retry it until it succeeds. Queues also provide useful features: buffering traffic spikes indepen