Study Library
17 min read
Event-Driven Systems•3,725 words

Inbox & Outbox Patterns

The two patterns that make event-driven systems correct rather than merely functional. Outbox is the write side (publish an event atomically with a state change). Inbox is the read side (process each event's effect at most once, even when it's delivered more than once). Together they convert an at-least-once transport (Kafka, RabbitMQ, SNS/SQS) into exactly-once effect end-to-end — which is the only "exactly-once" you can actually get in a distributed system.

Companion to kafka-eda-study-guide.md. That guide introduces the outbox at §8.3 and lists dual-write / duplicate processing as failure modes (§9.4, §9.8). This is the standalone deep dive on both patterns: JPA schemas, relay mechanics, the subtle correctness traps, and how they compose.

Code convention: every snippet is shown in Kotlin and Java (Spring Boot 3.x / Spring Data JPA / Spring Kafka, Jakarta namespace). Persistence is pure Spring Data — entities + repositories — with native SQL used in only two spots where JPQL genuinely can't express the operation; both are flagged and justified.

Organized as: the shared problem → Outbox in depth → Inbox in depth → composing them → alternatives → failure modes → interview framing.

1. The problem both patterns solve: the dual-write problem#

A service almost always needs to do two things when something happens: change its own state and tell the outside world.

Kotlin

kotlin
// The bug that looks like correct code @Service class OrderService( private val orderRepository: OrderRepository, private val kafkaTemplate: KafkaTemplate<String, OrderPlaced>, ) { @Transactional fun placeOrder(order: Order) { orderRepository.save(order) // write 1: our database kafkaTemplate.send("orders", order.id, OrderPlaced(order.id)) // write 2: the broker } }

Java

java
// The bug that looks like correct code @Service public class OrderService { private final OrderRepository orderRepository; private final KafkaTemplate<String, OrderPlaced> kafkaTemplate; // constructor omitted @Transactional public void placeOrder(Order order) { orderRepository.save(order); // write 1: our database kafkaTemplate.send("orders", order.id(), new OrderPlaced(order.id())); // write 2: the broker } }

These are two independent writes to two independent systems, and there is no transaction that spans both. Any crash, network blip, or timeout between them leaves the system inconsistent:

t0 BEGIN tx t1 INSERT order .................... committed to DB ✔ t2 << process crashes / broker unreachable / pod killed >> t3 kafka.send(OrderPlaced) ......... never happens ✗ ───────────────────────────────────────────────────── Result: order exists, but payment / inventory / email services never hear about it. Silent divergence.

The mirror image is just as bad — publish first, then the DB commit fails, and now there's a phantom event for an order that doesn't exist. Note that @Transactional above is a trap: it only wraps the database work. The kafkaTemplate.send fires outside the DB transaction's guarantees — worse, Spring flushes it before the transaction commits, so it can publish and then have the DB roll back underneath it.

Why not a distributed transaction (2PC/XA)? In theory an XA transaction spanning DB + broker is atomic. In practice: Kafka has no proper XA support; 2PC is a blocking protocol with the coordinator as a single point of failure; it serializes throughput and holds locks across a network round trip; and it fails badly under partitions. The industry verdict is to avoid 2PC and instead make a single atomic write, then propagate. That single atomic write is the outbox.

The two patterns attack the two halves of reliable propagation:

HalfPatternGuarantee it provides
Publish reliably alongside a state changeTransactional OutboxThe event is published if and only if the state change committed (at-least-once publish)
Consume without acting twice on redeliveryInbox / Idempotent ConsumerEach event's side effect happens at most once, even under duplicate delivery

Neither alone is sufficient. Outbox guarantees the event gets out but at-least-once, so duplicates are inevitable → you need the inbox. The inbox guarantees no double-processing but assumes the event was reliably published → you need the outbox. This is why they're taught as a pair.

2. The Transactional Outbox pattern (reliable publishing)#

Core idea: instead of writing to the DB and the broker, write only to the DB — but in the same local transaction, insert the event into an outbox table. Because both rows go to the same database, they commit atomically under ordinary ACID. A separate message relay then reads the outbox and publishes to the broker.

┌─────────────── one local DB transaction (atomic) ───────────────┐ │ save(order) -- business state │ │ save(outboxRow) -- "intent to publish" │ └─────────────────────────────────────────────────────────────────┘ │ commit ▼ ┌──────────────┐ reads unsent rows ┌────────┐ │ outbox table│ ─────────────────────▶ │ relay │ ──▶ Kafka └──────────────┘ marks sent / deletes └────────┘

The DB transaction makes "the order exists" and "we owe the world an OrderPlaced event" atomic. The event can never be lost once the business change commits, and can never be published for a change that rolled back.

2.1 The outbox entity#

Kotlin

kotlin
enum class OutboxStatus { PENDING, SENT } @Entity @Table(name = "outbox") class OutboxRecord( val aggregateType: String, // e.g. "Order" → routes to a topic val aggregateId: String, // e.g. order id → becomes the Kafka key val eventType: String, // e.g. "OrderPlaced" @JdbcTypeCode(SqlTypes.JSON) // Hibernate 6 maps this String to jsonb val payload: String, // the fully-serialized event (frozen snapshot) @Id val id: UUID = UUID.randomUUID(), // also the dedup / message id downstream @Enumerated(EnumType.STRING) var status: OutboxStatus = OutboxStatus.PENDING, val createdAt: Instant = Instant.now(), var sentAt: Instant? = null, )

Java

java
public enum OutboxStatus { PENDING, SENT } @Entity @Table(name = "outbox") public class OutboxRecord { @Id private UUID id = UUID.randomUUID(); // also the dedup / message id downstream private String aggregateType; // e.g. "Order" → routes to a topic private String aggregateId; // e.g. order id → becomes the Kafka key private String eventType; // e.g. "OrderPlaced" @JdbcTypeCode(SqlTypes.JSON) // Hibernate 6 maps this String to jsonb private String payload; // the fully-serialized event (frozen snapshot) @Enumerated(EnumType.STRING) private OutboxStatus status = OutboxStatus.PENDING; private Instant createdAt = Instant.now(); private Instant sentAt; protected OutboxRecord() {} // required by JPA public OutboxRecord(String aggregateType, String aggregateId, String eventType, String payload) { this.aggregateType = aggregateType; this.aggregateId = aggregateId; this.eventType = eventType; this.payload = payload; } // getters + status/sentAt setters omitted }

Design choices that matter:

  • Store the full serialized event, not just IDs. The relay publishes exactly what's in payload; it should never call back into business tables to "enrich" the event, because by publish time the aggregate may have changed again (you'd emit a state that never corresponded to this event). Serializing the payload inside the transaction freezes a correct snapshot. This also decouples the event schema from your table schema — you emit deliberate domain events, not a leaky dump of your columns.
  • id is a UUID and doubles as the downstream message id — that's the key the inbox dedups on later. It defaults at construction, inside the transaction.
  • aggregateId → partition key. Publishing with aggregateId as the Kafka key keeps all events for one aggregate in one partition, preserving per-entity ordering (see kafka guide §3).
  • @JdbcTypeCode(SqlTypes.JSON) (Hibernate 6) maps a String field to a Postgres jsonb column with no custom converter.
  • Index the unsent rows. The relay polls status = 'PENDING', so that predicate wants an index. JPA's @Index can only build a plain index; the ideal here is a partial index (CREATE INDEX ... ON outbox (created_at) WHERE status = 'PENDING') which JPA can't express — add it in your Flyway/Liquibase migration. It keeps the poll cheap even when millions of SENT rows remain.

2.2 The application write (the easy half)#

Kotlin

kotlin
@Service class OrderService( private val orderRepository: OrderRepository, private val outboxRepository: OutboxRepository, private val objectMapper: ObjectMapper, ) { @Transactional // ONE local transaction, DB only fun placeOrder(order: Order) { orderRepository.save(order) // business state outboxRepository.save( OutboxRecord( aggregateType = "Order", aggregateId = order.id, eventType = "OrderPlaced", payload = objectMapper.writeValueAsString(OrderPlaced(order)), // frozen snapshot ), ) } // commit → both rows or neither }

Java

java
@Service public class OrderService { private final OrderRepository orderRepository; private final OutboxRepository outboxRepository; private final ObjectMapper objectMapper; // constructor omitted @Transactional // ONE local transaction, DB only public void placeOrder(Order order) throws JsonProcessingException { orderRepository.save(order); // business state outboxRepository.save(new OutboxRecord( "Order", order.id(), "OrderPlaced", objectMapper.writeValueAsString(new OrderPlaced(order)))); // frozen snapshot } // commit → both rows or neither }

Notice there is no broker call here at all. The application's only job is to write the outbox row transactionally. All broker interaction moves to the relay. This is what makes it correct — there's nothing non-transactional left to fail.

2.3 The relay — flavor A: polling publisher#

A background job repeatedly claims a batch of unsent rows, publishes them, and marks them sent. The claim query is the crux: it must let multiple relay instances run without handing the same row to two of them. That's FOR UPDATE SKIP LOCKED, expressed in pure Spring Data JPA via @Lock plus a Hibernate lock-timeout hint:

Kotlin

kotlin
interface OutboxRepository : JpaRepository<OutboxRecord, UUID> { @Lock(LockModeType.PESSIMISTIC_WRITE) // → FOR UPDATE @QueryHints(QueryHint(name = "jakarta.persistence.lock.timeout", value = "-2")) // -2 = SKIP LOCKED @Query("SELECT o FROM OutboxRecord o WHERE o.status = :status ORDER BY o.createdAt") fun claimPending(@Param("status") status: OutboxStatus, pageable: Pageable): List<OutboxRecord> }

Java

java
public interface OutboxRepository extends JpaRepository<OutboxRecord, UUID> { @Lock(LockModeType.PESSIMISTIC_WRITE) // → FOR UPDATE @QueryHints(@QueryHint(name = "jakarta.persistence.lock.timeout", value = "-2")) // -2 = SKIP LOCKED @Query("SELECT o FROM OutboxRecord o WHERE o.status = :status ORDER BY o.createdAt") List<OutboxRecord> claimPending(@Param("status") OutboxStatus status, Pageable pageable); }

The magic number: Hibernate maps the lock-timeout hint value -2 to LockOptions.SKIP_LOCKED, 0 to NO_WAIT, and -1 to wait-forever. @Lock(PESSIMISTIC_WRITE) emits FOR UPDATE; the hint appends SKIP LOCKED. Together they are the pure-JPA spelling of the raw SQL claim. Pageable supplies the LIMIT.

Kotlin

kotlin
@Component class OutboxRelay( private val outboxRepository: OutboxRepository, private val kafkaTemplate: KafkaTemplate<String, String>, ) { @Scheduled(fixedDelay = 500) // poll every 500ms @Transactional // claim + mark-sent in one tx fun publishBatch() { val batch = outboxRepository.claimPending(OutboxStatus.PENDING, PageRequest.of(0, 100)) for (record in batch) { kafkaTemplate .send(topicFor(record), record.aggregateId, record.payload) // key = aggregateId .get() // await broker ack before marking SENT record.status = OutboxStatus.SENT // dirty-checked → UPDATE on commit record.sentAt = Instant.now() } } // rows released + statuses committed here }

Java

java
@Component public class OutboxRelay { private final OutboxRepository outboxRepository; private final KafkaTemplate<String, String> kafkaTemplate; // constructor omitted @Scheduled(fixedDelay = 500) // poll every 500ms @Transactional // claim + mark-sent in one tx public void publishBatch() throws Exception { List<OutboxRecord> batch = outboxRepository.claimPending(OutboxStatus.PENDING, PageRequest.of(0, 100)); for (OutboxRecord record : batch) { kafkaTemplate .send(topicFor(record), record.getAggregateId(), record.getPayload()) // key = aggregateId .get(); // await broker ack before marking SENT record.setStatus(OutboxStatus.SENT); // dirty-checked → UPDATE on commit record.setSentAt(Instant.now()); } } // rows released + statuses committed here }
  • .get() awaits the broker ack before flipping the row to SENT. If the publish fails it throws, the transaction rolls back, and the row stays PENDING for the next poll. (Blocking per-record is simplest and correct; for higher throughput, collect the send futures and await them as a batch before the status update.)
  • SKIP LOCKED lets you scale the relay horizontally — each instance grabs a disjoint batch, none blocks on another's locked rows.
  • Simple, portable, no extra infrastructure. The cost is polling latency (up to one interval) and steady query load. Tune the interval against load.

The subtle correctness trap — do NOT track a high-water-mark id. A tempting "optimization" is to drop the status column and instead remember the last id you published, querying WHERE id > :lastSeenId. This is broken because sequence / UUIDv7 ids are assigned when a row is created but become visible when its transaction commits, and transactions commit out of order:

T1: creates row A (earlier id), still running... T2: creates row B (later id), COMMITS first → relay sees B, sets lastSeen = B.id T1: COMMITS now (row A, earlier id) → relay never looks below B.id again ──────────────────────────────────────────────────────────────────────────────── Event A is silently never published. (a "gap" / lost event)

The fix is exactly the design above: claim by the status = PENDING flag, not by an id watermark. A late-committing row is still PENDING regardless of its id, so the next poll picks it up. (CDC, below, sidesteps this entirely by reading in commit order.)

2.4 The relay — flavor B: Change Data Capture (the modern default)#

Instead of polling, tail the database's transaction log (Postgres WAL / MySQL binlog) with Debezium and stream inserts to Kafka via Kafka Connect.

  • No polling, low latency, zero query load on your tables — it reads the log the DB already writes.
  • Reads in commit order, so the gap problem in §2.3 cannot occur.
  • Debezium ships a purpose-built Outbox Event Router SMT: point it at the outbox table and it routes each row to a topic derived from aggregateType and keys the record by aggregateId — the whole relay becomes connector configuration, not application code.
  • Costs: operational weight (Connect cluster, Debezium, connector tuning) and you must run and monitor it. For high-throughput systems it's worth it; for a single modest service, the polling relay above is often the pragmatic choice.

With CDC you can even delete() the outbox row in the same transaction that created it — Debezium still captures the INSERT from the log, and the table stays empty. (Neat, but verify your connector config captures the insert before relying on it.)

2.5 Delivery semantics & housekeeping#

  • Publishing is at-least-once. The relay can publish a row and then crash before the SENT status commits; on restart it republishes. This is unavoidable and by design — which is precisely why consumers need the inbox (§3). State this explicitly in an interview; claiming the outbox gives exactly-once is a red flag.
  • Ordering: the claim orders by createdAt, and keying by aggregateId gives per-entity order. Don't expect global ordering.
  • Table growth / cleanup: SENT rows accumulate. Either delete on publish, or keep them briefly for audit and batch-purge on a schedule (deleteByStatusAndSentAtBefore(...), or a @Query delete). An unbounded outbox becomes a hot, bloated table that drags down inserts and, on Postgres, stresses autovacuum. Treat retention as a deliberate decision.
  • Poison rows: a row that always fails to publish (e.g. a serialization bug) can wedge an ordered relay. Track an attempt count and move persistently failing rows aside for investigation rather than blocking the queue.

3. The Inbox pattern (idempotent consumer / deduplication)#

Core idea: the consumer records which messages it has already processed, and makes the "have I seen this?" check part of the same local transaction as the side effect. A redelivered message finds its id already recorded and is safely discarded.

Because the outbox (and every at-least-once broker) delivers duplicates, and because a consumer can crash after doing its work but before committing its offset, duplicates are not an edge case — they're guaranteed. Processing one twice means double charges, double emails, double inventory decrements. The inbox makes the effect idempotent.

3.1 The inbox entity#

Kotlin

kotlin
@Entity @Table(name = "inbox") class InboxMessage( @Id val messageId: UUID, // the outbox row's id, carried on the message val processedAt: Instant = Instant.now(), )

Java

java
@Entity @Table(name = "inbox") public class InboxMessage { @Id private UUID messageId; // the outbox row's id, carried on the message private Instant processedAt = Instant.now(); protected InboxMessage() {} // required by JPA public InboxMessage(UUID messageId) { this.messageId = messageId; } // getters omitted }

The entire mechanism is the primary key. That's the point — the database enforces "process once," not application logic.

3.2 The dedup insert — why it must be native ON CONFLICT#

The obvious "Spring" approach — if (inboxRepository.existsById(id)) return; else save(...) — is wrong: the check and the insert are two statements, so two concurrent deliveries of the same message both see "not present," both proceed, both act. The unique constraint must arbitrate atomically.

The next instinct — save(...) and catch DataIntegrityViolationException to detect the duplicate — is also wrong with JPA: a constraint violation marks the current transaction rollback-only, so you can't catch it and keep using the same transaction for the business effect. Catching it poisons the very transaction you need.

The correct tool is a native INSERT ... ON CONFLICT DO NOTHING that returns the affected-row count (1 = newly inserted, 0 = duplicate) and never throws on a conflict, leaving the transaction healthy:

Kotlin

kotlin
interface InboxRepository : JpaRepository<InboxMessage, UUID> { /** 1 = newly inserted, 0 = already processed (duplicate). Native because ON CONFLICT is * Postgres-specific (JPQL can't express it) AND because it must not throw — a caught * constraint violation would poison the surrounding JPA transaction. */ @Modifying @Query( value = "INSERT INTO inbox (message_id, processed_at) VALUES (:id, now()) " + "ON CONFLICT (message_id) DO NOTHING", nativeQuery = true, ) fun tryInsert(@Param("id") id: UUID): Int }

Java

java
public interface InboxRepository extends JpaRepository<InboxMessage, UUID> { /** 1 = newly inserted, 0 = already processed (duplicate). Native because ON CONFLICT is * Postgres-specific (JPQL can't express it) AND because it must not throw — a caught * constraint violation would poison the surrounding JPA transaction. */ @Modifying @Query(value = """ INSERT INTO inbox (message_id, processed_at) VALUES (:id, now()) ON CONFLICT (message_id) DO NOTHING """, nativeQuery = true) int tryInsert(@Param("id") UUID id); }

3.3 The consumer — dedup + effect in one transaction, ack afterward#

Two rules make this correct: (1) the dedup insert and the side effect commit in the same DB transaction; (2) the Kafka offset is committed only after that transaction commits. Rule 2 is why the listener acks outside the transactional method — so we split a thin @KafkaListener from a @Transactional service:

Kotlin

kotlin
@Component class InventoryListener( private val processor: OrderPlacedProcessor, ) { // container configured with ack-mode = MANUAL_IMMEDIATE, enable-auto-commit = false @KafkaListener(topics = ["orders"]) fun onMessage(event: OrderPlaced, @Header("messageId") messageId: UUID, ack: Acknowledgment) { processor.handle(event, messageId) // runs AND commits the DB transaction ack.acknowledge() // ONLY now commit the offset } } @Service class OrderPlacedProcessor( private val inboxRepository: InboxRepository, private val inventoryService: InventoryService, ) { @Transactional // dedup + effect in ONE tx fun handle(event: OrderPlaced, messageId: UUID) { if (inboxRepository.tryInsert(messageId) == 0) return // 0 → duplicate, skip the effect inventoryService.reserve(event.orderId) // business effect } // tx commits here }

Java

java
@Component public class InventoryListener { private final OrderPlacedProcessor processor; // constructor omitted // container configured with ack-mode = MANUAL_IMMEDIATE, enable-auto-commit = false @KafkaListener(topics = "orders") public void onMessage(OrderPlaced event, @Header("messageId") UUID messageId, Acknowledgment ack) { processor.handle(event, messageId); // runs AND commits the DB transaction ack.acknowledge(); // ONLY now commit the offset } } @Service public class OrderPlacedProcessor { private final InboxRepository inboxRepository; private final InventoryService inventoryService; // constructor omitted @Transactional // dedup + effect in ONE tx public void handle(OrderPlaced event, UUID messageId) { if (inboxRepository.tryInsert(messageId) == 0) return; // 0 → duplicate, skip the effect inventoryService.reserve(event.orderId()); // business effect } // tx commits here }

Why the ordering matters: crash after the DB commit but before the ack → Kafka redelivers → tryInsert returns 0 → the effect is safely skipped. The reverse order (ack first) would lose the message on a crash. Never call ack.acknowledge() inside the @Transactional method — with MANUAL_IMMEDIATE it commits the offset immediately, which could beat the DB commit and reopen the loss window.

3.4 What message id to dedup on#

  • Best: a stable business/event UUID carried in the message — for outbox-originated events this is the OutboxRecord.id, propagated as a Kafka header (messageId above). Survives topic/partition changes and replays.
  • Avoid using (topic, partition, offset) as the dedup key: offsets change if you reprocess from a new topic, mirror across clusters, or migrate — the "same" event gets a new coordinate and slips past dedup. Use an id that travels with the event's meaning, not its transport location.

3.5 Inbox retention#

You can't keep message ids forever — the table would grow without bound. Purge rows older than a redelivery horizon: the maximum age at which the broker could plausibly redeliver a message (broker retention, max consumer downtime, replay policy). A simple derived-query sweep on a schedule does it:

Kotlin

kotlin
@Modifying @Query("DELETE FROM InboxMessage m WHERE m.processedAt < :cutoff") fun purgeOlderThan(@Param("cutoff") cutoff: Instant): Int

Java

java
@Modifying @Query("DELETE FROM InboxMessage m WHERE m.processedAt < :cutoff") int purgeOlderThan(@Param("cutoff") Instant cutoff);

The risk is a too-short window: if you delete an id and that message is later redelivered (a long outage, a manual replay from an old offset), dedup misses and you double-process. Size the window against your worst-case replay, not the happy path.

3.6 Lighter-weight alternative: natural idempotency#

A dedicated inbox table is the general tool, but sometimes the operation is naturally idempotent and you don't need one:

  • Upsert on a business key — reprocessing converges to the same row:

    Kotlin

    kotlin
    @Modifying @Query(value = """ INSERT INTO order_projection (order_id, status) VALUES (:id, :status) ON CONFLICT (order_id) DO UPDATE SET status = EXCLUDED.status """, nativeQuery = true) fun upsert(@Param("id") id: String, @Param("status") status: String): Int

    Java

    java
    @Modifying @Query(value = """ INSERT INTO order_projection (order_id, status) VALUES (:id, :status) ON CONFLICT (order_id) DO UPDATE SET status = EXCLUDED.status """, nativeQuery = true) int upsert(@Param("id") String id, @Param("status") String status);

  • Absolute, not relative, writes: SET balance = :value is idempotent; balance = balance + :delta is not. Prefer set-to-value where the event carries the resulting state (pairs well with ECST events, kafka guide §8.1).

  • Idempotency key on the effect itself: e.g. a payments row with a @Table(uniqueConstraints = @UniqueConstraint(columnNames = "idempotency_key")) — the dedup and the business row are the same table, one fewer moving part.

Reach for the explicit inbox when the side effect isn't naturally idempotent (sending an email, calling a non-idempotent third-party API) or spans multiple rows.

3.7 What the inbox does not solve#

Dedup is not ordering. The inbox stops double-processing; it does nothing about out-of-order delivery (an updated arriving before its created). For that you still need per-key partitioning (kafka guide §3) and/or version/sequence checks in the handler that ignore stale updates. Say this explicitly — conflating "idempotent" with "ordered" is a common mistake.

4. Composing them — the reliable end-to-end pipeline#

The real power shows when a consumer is also a producer (the usual case in a chain of services). It does all three in one local transaction: dedup the incoming message, apply the effect, and write its own outbox row for whatever it needs to publish next. The thin @KafkaListener + ack wrapper from §3.3 is unchanged; only the transactional processor grows:

Kotlin

kotlin
@Service class OrderPlacedProcessor( private val inboxRepository: InboxRepository, private val outboxRepository: OutboxRepository, private val inventoryService: InventoryService, private val objectMapper: ObjectMapper, ) { @Transactional // inbox + effect + outbox: all atomic fun handle(event: OrderPlaced, messageId: UUID) { if (inboxRepository.tryInsert(messageId) == 0) return // duplicate → skip everything inventoryService.reserve(event.orderId) // business effect outboxRepository.save( // next event, same tx OutboxRecord( aggregateType = "Inventory", aggregateId = event.orderId, eventType = "InventoryReserved", payload = objectMapper.writeValueAsString(InventoryReserved(event.orderId)), ), ) } }

Java

java
@Service public class OrderPlacedProcessor { private final InboxRepository inboxRepository; private final OutboxRepository outboxRepository; private final InventoryService inventoryService; private final ObjectMapper objectMapper; // constructor omitted @Transactional // inbox + effect + outbox: all atomic public void handle(OrderPlaced event, UUID messageId) throws JsonProcessingException { if (inboxRepository.tryInsert(messageId) == 0) return; // duplicate → skip everything inventoryService.reserve(event.orderId()); // business effect outboxRepository.save(new OutboxRecord( // next event, same tx "Inventory", event.orderId(), "InventoryReserved", objectMapper.writeValueAsString(new InventoryReserved(event.orderId())))); } }
Service A Service B (consumer AND producer) ┌──────────────┐ Kafka ┌───────────────────────────────────────┐ Kafka │ business + │ ───────▶ │ inbox (dedup) + effect + outbox │ ───────▶ Service C … │ outbox (tx) │ │ ── all one local DB transaction ── │ └──────┬───────┘ └───────────────────┬───────────────────┘ │ relay │ relay └── at-least-once ──▶ duplicates ──▶ inbox absorbs them ──────┘

Chained across services, this is the backbone of a reliable saga (kafka guide §8.4): each step consumes idempotently and publishes transactionally, so the workflow survives crashes and redeliveries at every hop.

The exactly-once framing to state precisely: exactly-once delivery is impossible over a network (the two-generals problem — you can never be sure the ack arrived, so you must be willing to resend, so duplicates are always possible). What outbox + inbox achieve is exactly-once processing / effect: messages may be delivered many times, but each one's observable side effect happens once. "At-least-once delivery + idempotent consumption = exactly-once effect." That sentence is the whole point of both patterns, and saying it well is a strong senior signal.

5. Alternatives & how they compare#

ApproachWhat it doesWhen it's right / wrong
Transactional OutboxAtomic state change + intent-to-publish in one DB tx; relay publishesDefault for "DB write + must publish an event." Right nearly always for this need.
2PC / XADistributed transaction across DB + brokerAtomic in theory; blocking, coordinator SPOF, poor Kafka support, slow. Avoid.
Event SourcingThe event log is the source of truth — no separate state to dual-writeEliminates dual-write by construction, but a big commitment (kafka guide §8.1). Overkill if you just need to publish events.
CDC directly on business tablesDebezium streams row changes from orders etc.No outbox table, but your event schema = your table schema (leaky, couples consumers to internal columns; can't emit rich domain events or events that aren't a single row change). Outbox gives you deliberate events.
"Listen to yourself"Publish the event first, consume your own event to update the DBInverts source of truth to the log; niche, and reintroduces its own ordering/consistency puzzles.
Inbox / idempotent consumerDedup processed message ids in a local txDefault for the consume side. Use natural idempotency (§3.6) when the effect allows it.
Kafka EOS (transactions)Exactly-once for read-process-write inside KafkaGreat for Kafka→Kafka stream processing; does not cover your DB or a third-party API (kafka guide §2). For external effects you still need inbox/outbox.

6. Failure modes & fixes (Problem → Cause → Fix)#

6.1 Dual-write inconsistency — DB and broker disagree. Cause: two independent writes treated as atomic. Fix: Transactional Outbox; the application never calls kafkaTemplate.send alongside a DB write.

6.2 Lost outbox event via id-watermark polling — an event is never published. Cause: polling WHERE id > lastSeen while transactions commit out of order (§2.3). Fix: claim by a status = PENDING flag, not an id high-water mark; or use CDC (reads in commit order).

6.3 Duplicate published events — same event on the topic twice. Cause: relay crashes after publish, before the SENT status commits (inherent at-least-once). Fix: accept it — this is why consumers run the inbox (§3). Optionally enable the idempotent producer (kafka guide §4) to squash producer-retry dupes, but the relay-restart dup still needs consumer dedup.

6.4 Publish "succeeds" then DB rolls back — the row is marked SENT but wasn't really published, or vice versa. Cause: not awaiting the broker ack before the status update. Fix: .get() on the send future inside the relay transaction so a failed publish rolls the status back to PENDING.

6.5 Outbox table bloat — inserts slow down, autovacuum thrashes. Cause: SENT rows never removed. Fix: delete-on-publish or scheduled batch purge; partial index on unsent rows.

6.6 Stale event payload — a published event reflects a state that never existed. Cause: relay re-reads the aggregate at publish time, after it changed again. Fix: serialize the full payload inside the writing transaction; the relay publishes the stored string, never re-queries.

6.7 Double-processing despite an inbox — the effect still happens twice. Cause: existsById-then-save dedup race; or catching DataIntegrityViolationException (which marks the JPA tx rollback-only, so the effect never commits and redelivery repeats it); or the offset committed before the DB tx. Fix: native INSERT ... ON CONFLICT DO NOTHING returning a count (§3.2); one transaction for dedup + effect; ack only after that transaction commits.

6.8 Dedup misses on replay — an old message reprocesses. Cause: inbox retention window shorter than the redelivery/replay horizon (§3.5). Fix: keep message ids at least as long as the broker could redeliver or you'd replay; size for worst case.

6.9 Non-idempotent third-party effect — a duplicate still sends two emails / two charges. Cause: dedup protects your DB, but the external call isn't transactional with it. Fix: pass an idempotency key to the provider (Stripe et al. support this) so they dedup; or record "email sent for message X" in the same inbox transaction and check it before sending.

6.10 Out-of-order effects — updated applied before created. Cause: treating the inbox as an ordering mechanism. Fix: it isn't one — key by entity for per-partition order and/or use version checks to drop stale updates (§3.7).

6.11 Poison outbox row — relay wedged on one row. Cause: a row that always fails to serialize/publish blocks an ordered relay. Fix: attempt counter + move-aside/quarantine, alert, investigate; don't let one row halt the stream.

7. Interview framing — how to sound senior#

  • Trigger recognition is the whole game. The instant a design "writes to its database and publishes an event," say "that's a dual-write — I'd use the transactional outbox." The instant a consumer "might see the same message twice," say "make the consumer idempotent with an inbox / dedup key." Naming these unprompted is the highest-signal move.
  • Lead with the atomicity argument. The reason outbox works is that it collapses two writes into one local ACID transaction. If you can articulate why two systems can't be written atomically (and why 2PC is the wrong fix), you've shown you understand the root cause, not just the recipe.
  • Get the exactly-once nuance right. "Exactly-once delivery is impossible; outbox + inbox give exactly-once effect on at-least-once transport." Distinguish it from Kafka EOS (Kafka-internal only, not your DB or a third-party API).
  • Show the composition. The consumer-that-is-also-a-producer doing inbox + effect + outbox in one transaction is what makes an entire event-driven chain (a saga) reliable. That's the staff-level insight.
  • Talk trade-offs, not features: polling relay (simple, portable, latency + DB load) vs. CDC/Debezium (low-latency, no query load, operational weight); dedicated inbox table vs. natural idempotency (upsert / absolute writes / idempotency key); outbox vs. CDC-on-business-tables (deliberate domain events vs. leaky table-shaped events); retention windows on both tables.
  • Know the framework-level traps — the two places pure JPA leaks: FOR UPDATE SKIP LOCKED needs @Lock + the -2 lock-timeout hint, and inbox dedup needs a native ON CONFLICT because catching DataIntegrityViolationException poisons the transaction. Plus the design traps: id-watermark polling gap (§2.3), payload staleness (§6.6), dedup vs. ordering (§3.7). Any one of these separates people who've run these patterns from people who've only read about them.
  • Attribution that lands: Chris Richardson codified Outbox, Inbox/Idempotent Consumer, and Saga at microservices.io; Debezium's Outbox Event Router is the reference CDC implementation; Spring's own guidance is to prefer at-least-once + idempotent consumers over chasing exactly-once delivery.

Quick self-test: Why can't @Transactional around orderRepository.save + kafkaTemplate.send fix the dual-write? Why does the outbox still need idempotent consumers? Why is WHERE id > lastSeen a broken way to poll the outbox? Why must inbox dedup use native ON CONFLICT rather than catching DataIntegrityViolationException? Why must ack.acknowledge() run outside the @Transactional method, and after it? Why is "exactly-once delivery" the wrong phrase, and what's the right one? What do you do when the duplicated effect is an external email send? If you can answer all seven, you're ready.