Outbox Pattern: Reliable Event Publishing

What it is

The outbox pattern solves the dual-write problem: how can an event be reliably published after writing to the database, given that the database and the message broker are independent systems?

The idea: don't try to write to two systems atomically. Write the event to a table in the same database, in the same transaction as the business write. A separate process reads the table and publishes events to the broker. Now there is one atomic write (the transaction), and the publishing is decoupled from the request path.

This pattern shows up in every production system that publishes events. Outbox, transactional outbox, transactional messaging, same idea, different names.

Why dual-write fails

An order needs to be created: insert into the orders table AND publish an OrderCreated event to Kafka. Two writes, two systems, no atomicity.

Failure modes:

1. DB succeeds, broker fails. Order exists; nobody knows about it. Customer sees confirmation; no email is sent, no fulfilment kicks off.
2. Broker succeeds, DB fails. Phantom event. Consumers process an order that doesn't exist.
3. DB succeeds, broker partially fails. Some downstream subscribers get the event; others don't.

There's no order of operations that fixes this. The outbox pattern recognises that two systems can't be made atomic, so it makes the operation atomic in one system (the DB), then handles publishing as a separate, retryable step.

The relay

The relay is a separate process (or thread, or scheduled job) that polls the outbox table and publishes pending rows. The standard implementation:

loop:
  in transaction:
    select pending rows for update skip locked, limit 100
    for each row: publish to broker
    mark rows as published
    commit
  sleep briefly if no rows

SKIP LOCKED lets multiple relay instances run safely; each grabs a different batch.

The relay is at-least-once by design. If it crashes between publishing and marking-sent, those events get republished on restart. Consumers must be idempotent. Combine with idempotency keys to make this safe.

Outbox vs CDC

Two ways to get changes from the database into the event stream.

Outbox: explicit table. The event payload is written at the time of the business write. Clean event design (the schema is under application control). Extra DB table, extra relay process.

CDC (Change Data Capture): tools like Debezium read the database's WAL and publish each change as an event. No outbox table needed. Every change becomes an event automatically. Trade-offs: less control over the event shape (it's the row), publishes changes that may not be intended as events, requires platform-specific setup (Postgres logical replication, MySQL binlog).

For greenfield: pick based on whether event-design control (outbox) or zero application code overhead (CDC) matters more. Many teams use both: CDC for cross-team event streaming, outbox for application-level workflow events.

Performance considerations

For high-volume systems, the outbox can become hot:

• Index (published_at IS NULL, created_at) for the relay query.
• Partition by date if the table grows.
• Delete published rows (or move to an archive) to keep the active set small.
• Run multiple relay instances with SKIP LOCKED to scale throughput.

Relay latency = poll interval (usually 100ms to a few seconds). For lower latency, consider Postgres LISTEN/NOTIFY to wake the relay on insert.

The outbox INSERT must commit in the same transaction as the business write. Without that, the whole point is gone. Consumers must be idempotent because outbox delivery is at-least-once. And the relay becomes an operational responsibility: monitor lag (how many pending rows, how old) and alert on a growing backlog.

For Java, libraries like Spring's transactional event listener with the outbox extension simplify this. For Go, libraries like Watermill or a hand-rolled relay are common. For Python, custom code or workflow engines (Temporal handles this implicitly).