Background Job Queue

What it is

A background job queue moves work out of the request path and into asynchronous workers. The user clicks "send", the request handler enqueues a job and responds immediately. A worker picks up the job, sends the email, marks it done.

The pattern is everywhere: email sending, image processing, batch reports, webhooks, daily/hourly jobs. The infrastructure pieces are the same: durable queue, workers, retries, dead-letter handling.

What it requires

Durable storage. Jobs must survive process and machine restart. In-memory queues lose work on crash. Real options: Postgres SKIP LOCKED, Redis (with persistence and consumer groups), AWS SQS, RabbitMQ, Kafka.

Visibility timeout. When a worker claims a job, the queue hides it for some duration (typically 30-300s). If the worker doesn't ack/delete within that window, the job reappears for another worker. Handles worker crashes automatically.

At-least-once delivery + idempotent handlers. Visibility timeout means jobs can be delivered twice (worker timed out, but actually finished, just didn't ack). Handlers must tolerate this: idempotency key check, transactional "do + record done", or naturally idempotent operations.

Retry with backoff. Failed jobs are re-queued with increasing delay. Same exponential backoff with jitter as HTTP retries. Cap the attempts.

Dead-letter queue. After max attempts, the job moves to a separate "dead" table for human inspection. Without DLQ, persistently failing jobs retry forever and flood the queue.

The Postgres SKIP LOCKED pattern

For teams that already run Postgres, this avoids adding a new system:

UPDATE jobs SET state = 'processing', claimed_at = now()
WHERE id = (
    SELECT id FROM jobs
    WHERE state = 'pending' AND run_at <= now()
    ORDER BY priority DESC, run_at
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING id, payload;

Each worker selects one unlocked job, locks it, takes it. Others SKIP the locked rows. Throughput scales with Postgres. Transactional with the rest of the application's data.

For higher throughput or more complex scheduling, dedicated queue systems (Sidekiq for Ruby, Celery for Python, BullMQ for Node, SQS for AWS) give better tooling and observability.

Why idempotency matters here

Visibility-timeout-based queues guarantee at-least-once delivery, not exactly-once. The flow:

1. Worker A claims job 42, starts processing.
2. Worker A successfully completes the work (charges the card).
3. Worker A is killed before it can delete the job.
4. Visibility timeout expires.
5. Worker B claims job 42.
6. Worker B charges the card again.

Without idempotency on the handler, the customer sees two charges. With idempotency (check "have I processed job 42?" before charging), worker B's attempt is a no-op.

What every system gets wrong

Three common mistakes:

No DLQ. Failed jobs retry forever. The queue grows. Real failures are buried in noise. Always have a DLQ and an alert on its size.

Non-idempotent handlers. "It only happens occasionally" is what people say right before the duplicate-charge incident. Make handlers idempotent from day one.

Treating the queue as a database. Long-running queries against the queue table; transactions held for the whole job duration; row-level locks competing with workers. Keep job processing fast (or move long work into the worker process, not held in the queue transaction).