The Saga Pattern

Distributed transactions without a distributed lock. A reference for system-design and deep-dive interviews. Organized as: mental model → why not 2PC → anatomy → recovery taxonomy → the two coordination types → isolation → correctness → related patterns (TCC, routing slip) → tooling → problems & fixes → interview framing → cheat-sheet → self-test.

The saga is first a concurrency-control idea and only second a microservices idea. Almost every hard question about it ("what happened to isolation?", "why can't I just roll back?") comes from that first identity. Keep it in mind and the rest follows.

1. The mental model#

A saga is a sequence of local transactions T1, T2, … Tn, each running in a different service (and its own database), coordinated so that the whole thing either completes, or is semantically undone by running compensating transactions Cn-1, … C1 for the steps that already committed.

There is no global transaction, no shared lock, no coordinator holding everyone hostage. Each Ti commits independently and immediately in its own service. Atomicity across the whole business operation is simulated after the fact: if step k fails, you don't roll back T1…Tk-1 (they're already committed and durable) — you run C1…Ck-1 to compensate for them.

The one-line framing for interviews: "A saga replaces one distributed ACID transaction with a sequence of local ACID transactions plus compensating transactions that semantically undo the committed ones on failure. You trade isolation and instant rollback for availability and loose coupling — the result is eventually consistent."

Origin (worth knowing, occasionally asked)#

The term comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem ("Sagas", Princeton). Their problem had nothing to do with microservices — it was long-lived transactions (LLTs) inside a single database: a transaction that runs for minutes or hours (a batch, a report, a booking) holds locks the whole time and destroys concurrency. Their fix: break the LLT into a chain of short local transactions that each commit and release locks immediately, and give each one a compensating transaction so the chain can be logically undone. The database releases locks between steps, so other work proceeds.

The modern microservices usage repurposes that idea: the "long transaction that can't hold locks" becomes "a business operation spanning services that can't share a transaction at all." Same structure (local steps + compensations), different reason (physical impossibility of a shared transaction across service-owned databases, not just lock contention).

ACD, not ACID — the defining property#

A saga gives you Atomicity (all-or-nothing, but semantically, via compensation), Consistency, and Durability — but not Isolation. There is no point at which the whole operation is invisible-until-committed. Intermediate states are real, committed, and visible to everyone else while the saga is still in flight.

This missing "I" is not a footnote — it is the source of essentially every subtle saga bug (§9) and every countermeasure you'll design (§9 countermeasures). Anyone who describes a saga without saying "you lose isolation" doesn't really understand it.

2. Why not just use 2PC / XA?#

The honest framing: a saga is what you reach for because you refuse to hold a distributed transaction. So you must be able to say crisply why 2PC is off the table.

Two-Phase Commit (2PC / XA) is the classical distributed-transaction protocol. A transaction coordinator runs two rounds:

• Prepare (vote) phase: coordinator asks every participant "can you commit?" Each participant does the work, writes it to durable storage, takes and holds locks, and votes yes/no — but does not commit yet.
• Commit phase: if everyone voted yes, coordinator tells all to commit; if anyone voted no, it tells all to abort.

It genuinely gives you ACID across services. So why is it avoided in microservices?

• It blocks, holding locks across the network. Between prepare and commit, every participant sits with rows/resources locked, waiting on the coordinator. Latency and lock-hold time are now a function of the slowest participant and the network. Throughput collapses under contention.
• The coordinator is a single point of failure — and worse, 2PC can get stuck. If the coordinator dies after participants voted "yes" but before the commit/abort decision reaches them, participants are in-doubt: they hold locks and cannot unilaterally decide. This is the classic 2PC blocking problem. (3PC and Paxos-Commit reduce but don't eliminate the operational pain.)
• It's CP, not AP. By CAP, 2PC chooses consistency over availability: a partition or a down participant stalls the whole transaction. Microservices usually prefer to stay available and reconcile.
• Much of the modern stack doesn't support XA. Kafka, most NoSQL stores, most REST/gRPC services, and many managed cloud services have no XA transaction manager. You often cannot enlist them in a distributed transaction even if you wanted to.
• It couples deploy/runtime lifecycles. Everyone must be up, reachable, and XA-capable at the same instant.

So the trade is explicit: 2PC keeps isolation and instant rollback but sacrifices availability, throughput, and stack-freedom. A saga keeps availability and loose coupling but sacrifices isolation and gives you only eventual, compensation-based "rollback."

A strong senior line: "Before I reach for a saga, I ask whether the transaction should span services at all — often the right fix is better bounded-context design so one service owns the whole invariant and a single local ACID transaction does the job. Saga is for when the process is genuinely distributed." Distributed transactions are a cost; the best one is the one you designed away.

3. Anatomy of a saga#

Four moving parts. Get the vocabulary exact.

Local transaction (Ti). A normal ACID transaction inside one service's own database. It does the service's piece of work and, atomically with it, records the fact that it happened (usually by emitting an event or a reply — see the outbox in §10.1, because that "atomically" is not free).

Compensating transaction (Ci). A local transaction that semantically undoes the effect of Ti. Key word: semantically. It is a new forward transaction, not a database rollback.

• You can't "un-charge" a card — you issue a refund (a new record). The audit trail shows charge then refund, not nothing.
• You can't "un-send" an email — you send a correction/apology.
• You can't "un-decrement" inventory by rolling back — you run a compensating increment (restock).
• Compensation may not restore byte-for-byte the prior state, and that's expected. What it restores is the business invariant ("the customer isn't out any money"), not the literal bytes.

Saga log / saga state. Durable record of where the saga is: which steps have run, which succeeded, whether it's going forward or compensating. Without it, a crash mid-saga is unrecoverable — you'd have committed steps and no memory that they belong to an incomplete operation. In orchestration this lives in the orchestrator's store; in choreography it's distributed across each participant's state (harder to see — a real drawback, §7).

Messaging / transport. Events (choreography) or commands + replies (orchestration), almost always over durable async messaging (Kafka, RabbitMQ, SQS). Because delivery is at-least-once, every handler and every compensation must be idempotent (§10.2).

Three ironclad rules for compensations#

1. Compensations must be idempotent. They get retried (at-least-once delivery, crash-and-resume). "Refund order 42" applied twice must not refund twice.
2. Compensations must (essentially) not fail — or must be retried until they succeed. A failed compensation leaves the system inconsistent. You retry with backoff and, past a limit, escalate to a human / alert on a "stuck saga." There is no compensation-for-a-compensation turtles-all-the-way-down; the buck stops with retry-then-alert.
3. Compensate in reverse order of completion (Ck-1 … C1), because later steps may depend on earlier ones.

4. Recovery taxonomy: backward, forward, and the pivot#

"All types of saga" is partly about coordination (§5–6: choreography vs orchestration) and partly about recovery direction and step classification. This section is the second axis.

Backward vs forward recovery#

• Backward recovery (compensation). On failure, undo completed steps by running their compensations. This is the "classic" saga. Used when a step can be undone and undoing is the right business response.
• Forward recovery (retry to completion). On failure, don't undo — keep retrying the failed step (and continue forward) until it succeeds. Used when the step must eventually happen and undoing prior work is undesirable or impossible. Requires steps to be retriable (idempotent, and guaranteed to eventually succeed given enough retries — e.g., "send the confirmation email", "credit the destination account").

Real sagas mix both: compensate the early part, but once you pass a certain point, only go forward.

Chris Richardson's transaction classification (the high-signal bit)#

Every step in a well-designed saga is one of three kinds, and their order matters:

• Compensatable transactions — steps that can be semantically undone. They come first.
• Pivot transaction — the point of no return / go-no-go. Once the pivot commits, the saga will run to completion (forward only); if the pivot aborts, the saga fails and everything before it is compensated. The pivot is neither compensatable nor retriable — it's the hinge. (Often it's the last step that could still be compensated, or the first that is only retriable, or a genuinely irreversible action like "capture non-refundable payment".)
• Retriable transactions — steps that come after the pivot and are guaranteed to succeed with retries. They need no compensation, because the saga is already committed to going forward.

Ordering principle: arrange the saga as [compensatable…] → pivot → [retriable…]. Put the reversible, risky, likely-to-fail work before the pivot (so failure is cheap — just compensate); put the must-happen, safe-to-retry work after it. This minimizes the compensation window and means that once you're past the pivot you never have to unwind.

Concretely, in an order saga: reserving credit and reserving inventory are compensatable (release them if something later fails); approving/placing the order (the customer is now committed) is a natural pivot; sending the confirmation and notifying the warehouse are retriable (they must happen, retry until they do, never worth undoing).

Naming the pivot in an interview signals you actually understand saga design rather than just "choreography vs orchestration."

5. Type 1 — Choreography-based saga (event-driven, no coordinator)#

Idea: there is no central brain. Each service subscribes to events and reacts: it does its local transaction and publishes an event, which some other service is listening for, which does its work and publishes the next event, and so on. The workflow is emergent — it exists only as the sum of everyone's subscriptions. Nobody "runs" the saga; it happens.

The happy path (e-commerce order)#

Each arrow is a local ACID transaction + an event. The event is what triggers the next hop.

The failure/compensation path#

Say inventory is out of stock. Inventory publishes a failure event and the chain runs backward:

Notice compensation is also just event-driven choreography, walking back through the services that had already committed.

Code sketch — Payment Service (Spring Boot + Spring Kafka)#

The participant just reacts to events and emits its own; there is no saga class. Note idempotency and the outbox in every handler (including the compensation).

Kotlin

Java

There is no OrderSaga class anywhere. That's the point — and the problem.

Pros#

• Loose coupling, no central bottleneck or SPOF. Services only know about events, not about each other or an orchestrator.
• Easy to add a participant for new reactions: a new "Loyalty Service" can subscribe to PaymentCaptured and award points without touching anyone else.
• Minimal moving parts — no orchestrator to build, deploy, and operate. Great for short, stable flows (≈2–4 steps).

Cons#

• No one place shows the end-to-end flow. The business process is scattered across N services' listeners. To answer "what happens when an order is placed?" you must read every service. This is the dominant complaint and it worsens fast with step count.
• Hard to debug and monitor. There's no saga state to look at; you reconstruct flows from distributed traces/logs. "Where is order 42 stuck?" has no single answer.
• Risk of cyclic event dependencies — A emits an event B reacts to by emitting one A reacts to… accidental infinite loops and tangled graphs.
• Changing the flow touches many services. Insert a "fraud check" between payment and inventory and you must rewire several services' subscriptions and compensations.
• Testing needs the whole cast. Meaningful tests span every participant.

Use choreography when: few steps, stable process, high autonomy desired, and the flow is simple enough to hold in your head.

6. Type 2 — Orchestration-based saga (a coordinator drives it)#

Idea: a single orchestrator (a.k.a. Saga Execution Coordinator, or in EIP terms a Process Manager) owns the workflow as an explicit state machine. It sends a command to a participant ("reserve credit"), waits for the reply ("credit reserved" / "credit failed"), decides the next command, and persists its state after every transition. Participants are dumb: they execute a command and reply. They don't know each other or the flow.

The happy path (same order, orchestrated)#

The failure path — orchestrator issues compensating commands in reverse#

The orchestrator knows exactly which steps have completed, so it knows exactly which compensations to send and in what order. Compensation logic is centralized and explicit — a big reason orchestration scales to complex flows.

Code sketch — an explicit, persisted state machine (Spring Boot)#

The orchestrator persists its state after every transition (that persisted row is the saga log, §3), consumes participant replies, and drives the next command — or, on a failure before the pivot, the compensating commands in reverse. A sealed reply type lets the compiler force you to handle every branch.

Kotlin

Java

Frameworks (Eventuate Tram Sagas, Axon, Temporal, Camunda) give you this skeleton — a saga DSL / state machine plus durable state, timeouts, and retries — so you don't hand-roll persistence and recovery. Temporal goes furthest: you write the workflow as ordinary sequential code and the engine transparently persists every step and replays on failure (durable execution), so a saga looks like a try/catch that runs its registered compensations on failure (this uses Temporal's built-in io.temporal.workflow.Saga helper):

Kotlin

Java

Pros#

• The workflow is explicit and in one place. You can read the whole business process, draw it, reason about it, and change it centrally.
• Best for complex flows — many steps, conditional branches, parallel steps, many compensations.
• Superior observability & operations — the orchestrator holds saga state, so "where is order 42?" is a single lookup; timeouts, retries, and stuck-saga detection live in one place.
• No cyclic dependencies; participants are decoupled from each other (they only know the orchestrator).
• Easier evolution — insert/reorder steps by editing the orchestrator, not every service.

Cons#

• The orchestrator is extra infrastructure to build/operate (or a product to adopt).
• Risk of a "god orchestrator." Business logic that belongs in services leaks into the orchestrator, leaving anemic services. Keep the orchestrator about sequencing and compensation, not domain rules.
• A coupling/throughput focal point. It's on the path of every saga. Mitigate by making it durable and horizontally scalable (partition by sagaId) rather than a singleton — modern engines do this for you, so it is not a true SPOF when built right.
• More upfront design than "just subscribe to an event."

Use orchestration when: many steps, complex compensation, conditional/parallel logic, or you need first-class visibility and operational control (which is most non-trivial business processes).

7. Choosing between them (and hybrids)#

Rule of thumb: choreography for simple, stable, few-step flows where autonomy matters; orchestration the moment the process has many steps, real compensations, timeouts, or branching you need to see and control. When in doubt for a business-critical multi-step process, most teams pick orchestration for the observability alone.

Hybrid is legitimate and common. Use orchestration within a bounded context / complex sub-flow, and choreography between contexts (emit a domain event at the boundary; the next context's orchestrator picks it up). This keeps each complex process observable while keeping contexts loosely coupled. Don't treat the two as a religious either/or.

8. Worked example — one order saga, both styles#

Same business process, so you can see the difference is purely coordination. Steps, states, and compensations:

• Choreography: steps 1→6 each publish an event the next service listens for; a failure at 2 or 3 publishes a failure event that walks compensations back to 1. After step 4 (pivot), 5 and 6 only ever go forward (retry on failure), never compensate.
• Orchestration: one CreateOrderSaga sends commands 1→6 and, on a failure before the pivot, sends compensating commands in reverse (release inventory, then release/refund credit, then cancel order). After the pivot it retries 5 and 6 until they succeed.

The pivot at step 4 is why 5 and 6 are safe to design as retriable: once the customer's order is approved and paid, "send the email" and "tell the warehouse" must happen and are never worth undoing — so you retry, you don't compensate.

9. The isolation problem (the missing "I") and its countermeasures#

This is the deepest part of saga design and the strongest senior/staff signal. Because a saga is ACD, not ACID (§1), every intermediate state is committed and visible. Other sagas and ordinary transactions can read and write that in-flight data. That produces the classic concurrency anomalies — the same ones isolation levels exist to prevent in a database, except now you have no isolation and must handle them at the application level.

The anomalies (know all three by name)#

• Lost updates. Saga A updates a record; saga B (or a plain transaction) overwrites it before A finishes; A's later step or compensation is now based on stale data, or B's write silently disappears. Example: saga A reserves the last unit of stock; concurrently B reads "1 available" (A hadn't decremented yet in B's view) and also sells it → oversold.
• Dirty reads. A transaction reads data a saga wrote before the saga completes — and the saga later compensates, so the read was of a value that "never should have existed." Example: an order sits in APPROVED mid-saga, a downstream report counts it as revenue, then the saga fails and compensates to CANCELLED → the report double-counted a cancelled order.
• Fuzzy / non-repeatable reads. A saga reads the same record at step 2 and again at step 5 and gets different values because another saga modified it in between → the saga makes a decision on inconsistent data.

The countermeasures (Chris Richardson's names — memorize these)#

These raise the effective isolation of a saga without a distributed lock. You pick per-anomaly, per-risk.

• Semantic lock. The application-level lock and the workhorse. A compensatable transaction sets a flag / lock state on any record it touches — e.g. an order status of *_PENDING (APPROVAL_PENDING), or a lockedBy = sagaId column. Other sagas that encounter the flag must react: block/retry later, fail fast, or take a defined alternate path. The saga's final step (completion or compensation) clears the lock. This is a mutex expressed in business state; it directly defends against dirty reads (readers see "pending, don't trust me yet") and serializes conflicting sagas. Cost: you must design the pending states and teach every reader/writer to honor them, and handle deadlock/timeout on the lock.
• Commutative updates. Design updates so order doesn't matter — if operations commute, you can't lose an update by applying them in a different order. debit(100) and credit(30) on a balance commute (final balance is the same either way); an absolute set(balance, X) does not commute. Prefer relative/commutative operations (increment, add, append) over absolute overwrites. Defends against lost updates.
• Pessimistic view. Reorder the saga's steps so that the steps most exposed to dirty reads run late (or the risky read runs after the risky write is finalized), trading a little forward-progress for less exposure to reading uncommitted-then-compensated data. It's "sequence the saga to minimize business risk from the missing isolation." Defends against dirty reads by construction rather than by locking.
• Reread value (a.k.a. optimistic offline lock / version check). Before a write, re-read the record and verify it hasn't changed since you last read it (compare a version/timestamp column, i.e. optimistic concurrency). If it changed, you've hit a conflicting update → abort/retry the saga instead of clobbering. Defends against lost updates cheaply, without holding a lock.
• Version file. Record each operation on a record as an entry in a log ("version file") instead of applying it destructively. Because you keep the operations, you can reorder/reconcile out-of-order operations after the fact — effectively turning non-commutative operations into commutative ones by replaying them in the right order. Defends against out-of-order / lost updates when operations arrive in an unpredictable sequence (very common with async messaging).
• By value. A meta-strategy: choose the concurrency mechanism per request based on business risk. Low-risk, low-value requests use a plain saga (accept the weak isolation); high-risk, high-value requests are routed to a stricter mechanism (e.g. an actual distributed transaction, or a saga with heavy semantic locking). You dynamically dial isolation up where the money/impact justifies the cost.

How to deploy them: most systems lean on semantic locks for the "don't act on my in-flight state" problem, reread/optimistic version checks for lost updates, and commutative updates where the domain allows it, reserving version files and by-value for the genuinely hard cases. The interview-grade sentence: "A saga has no isolation, so I add it back selectively at the application layer — semantic locks for dirty reads, optimistic version checks and commutative updates for lost updates, and step reordering (pessimistic view) to shrink the exposure window — rather than pretending intermediate states aren't visible."

In code — semantic lock + optimistic version (Spring Boot + JPA). The *_PENDING status is the semantic lock; the @Version column is the reread-value countermeasure — JPA re-checks it on flush and throws if another transaction changed the row.

Kotlin

Java

10. Correctness essentials (the things that actually break in production)#

A saga is only correct if each of these is handled. Interviewers probe them because they're where real systems fail.

10.1 The dual-write problem inside every step → Transactional Outbox#

Each step must update its database and emit its event/command/reply atomically. If it does db.save() then broker.send() as two separate calls, a crash between them either loses the event (DB changed, nobody notified — the saga stalls forever) or emits without the DB change (phantom step). You cannot atomically write to a database and a message broker with two independent calls — there is no shared transaction.

Fix — Transactional Outbox. In the same local DB transaction as the business write, insert a row into an outbox table. A separate relay publishes those rows and marks them sent. The DB transaction makes "the state change" and "the intent to publish" atomic. Run the relay via CDC (Debezium tailing the transaction log) — the modern default — or a polling publisher. Publishing is at-least-once (the relay can crash after publish, before marking sent), which is exactly why the next point is mandatory. This is the pattern that glues saga steps to reliable messaging; a saga without it has a correctness hole in every step.

In code — one local transaction writes both (Spring Boot + JPA). The business row and the event row share a single @Transactional boundary; nothing publishes to Kafka inline.

Kotlin

Java

10.2 Idempotency everywhere (because delivery is at-least-once)#

Every command handler, every event handler, and every compensation must be idempotent — processing the same message twice must have the same effect as once. Implement with a dedup/idempotency key (the message ID or a business key) checked against a processed_messages table with a unique constraint, an upsert, or a Redis seen-set with TTL — or make the operation naturally idempotent (SET status=REFUNDED not balance -= x unless guarded). "We'll just make sure it's delivered once" is a red-flag answer; design for duplicates.

In code — the markIfNew dedup the handlers in §5 call (Spring Boot + JPA). The primary key gives the uniqueness; a clash means "already processed."

Kotlin

Java

10.3 Compensation must eventually succeed#

Compensations fail too (network, downstream down). Because a failed compensation leaves the system inconsistent, you retry with exponential backoff + jitter, and past a bound, escalate: alert, park the saga in a COMPENSATION_FAILED state, and let an operator intervene. Compensations should be built to be as reliable and cheap as possible (ideally local, no fan-out). There is no automatic "compensate the compensation" — the terminal strategy is retry-then-human.

10.4 Timeouts and the "stuck saga"#

A participant may never reply (crash, lost message, poison pill). Without timeouts the saga hangs forever, holding semantic locks (§9) and blocking others. Every orchestrated step needs a timeout; on expiry the orchestrator decides: retry the command, or treat it as failure and compensate. In choreography, there's no natural owner of the timeout — you need a separate watchdog/scheduler per step, which is one more reason choreography gets unwieldy. Detecting and resolving stuck sagas (timed-out, or compensation-failed) is a first-class operational concern: emit metrics on saga age and state, alert on sagas older than SLA.

10.5 Saga state must be durable and recoverable#

The saga log (§3) must survive a crash so the coordinator (or the choreography participants) can resume: re-send the in-flight command, or continue compensating. Orchestration engines (Temporal, Camunda, Eventuate) persist this for you; hand-rolled sagas must persist (sagaId, state, step, direction) transactionally with each transition — again via the outbox so the state change and the next command are atomic.

11. Related / alternative patterns: TCC and Routing Slip#

"All types" should include the two patterns that sit right next to sagas — one improves isolation, one is a decentralized-with-itinerary hybrid.

11.1 TCC — Try / Confirm / Cancel (reservation-based)#

TCC is a distributed-transaction pattern often discussed as a saga variant with better isolation. Each participant exposes three operations instead of one:

• Try — reserve the resources / do the tentative work, without making it final or publicly visible. (Reserve inventory into a "held" bucket; authorize — not capture — the card.)
• Confirm — make the reserved work permanent. Called only if all participants' Try succeeded. Must be idempotent.
• Cancel — release the reservation (the compensation for Try). Called if any Try failed. Must be idempotent.

A coordinator runs Try on all participants; if all succeed → Confirm all; if any fails → Cancel all.

TCC vs plain saga — the key distinction: in a plain saga, each step fully commits and is immediately visible, so intermediate states cause dirty reads (§9). In TCC, the Try phase holds resources in a reserved / invisible state, so other transactions don't see them as available — TCC has an explicit reservation phase and therefore better isolation (no dirty reads of the reserved resource). It's like 2PC's prepare phase, but implemented in the application with business semantics and no held database locks, so it doesn't block like 2PC. The cost: every participant must implement three well-behaved, idempotent methods and manage reservation lifetimes/expiry. Think of TCC as "reservation-based saga" — more work, more isolation.

In code — the three-method participant contract (Spring Boot). Every participant implements the same shape; confirm and cancel must be idempotent and keyed by txId.

Kotlin

Java

11.2 Routing Slip (itinerary travels with the message)#

A decentralized pattern (from Enterprise Integration Patterns; MassTransit's Courier implements it) that gets you saga-like behavior without a central orchestrator and without choreography's implicit flow. The route — the ordered list of steps and each step's compensation — is attached to the message as an itinerary. Each service executes its activity, records its compensation onto the slip, and forwards the message to the next step in the itinerary. On failure, the accumulated compensations on the slip are executed in reverse. So the workflow is explicit and self-describing (it's in the message) yet executed in a distributed way (no orchestrator service) — a genuine middle ground between the two main types.

12. 2PC vs Saga vs TCC — the comparison to have ready#

13. Tooling landscape (name these; know what class each is)#

Durable-execution / workflow engines (orchestration):

• Temporal (and its ancestor Cadence) — durable execution: you write the saga as ordinary sequential code; the engine persists every step and replays on failure, so state, retries, and timeouts are handled for you. Very popular modern default for orchestrated sagas. Compensation is just code you run in a catch/finally.
• Camunda 8 / Zeebe — BPMN-based workflow engine; Zeebe is the cloud-native, horizontally scalable core. You model the process (including compensation events) as a diagram. Strong when business stakeholders want to see the process. (Camunda 7 is the older embedded/JVM engine.)
• Netflix Conductor (and managed Orkes) — JSON-DSL orchestrator for microservice workflows.
• AWS Step Functions — serverless state-machine orchestrator; AWS documents an explicit saga implementation (Step Functions coordinating Lambdas with compensation branches). Good default on AWS.
• Others you can name: Azure Durable Functions, Uber Cadence (open-source Temporal predecessor), Apache Airflow (data pipelines, sometimes stretched into orchestration).

Saga/messaging frameworks (library-level):

• Eventuate Tram Sagas (Chris Richardson) — Java/Spring; an orchestration saga DSL plus transactional messaging (outbox/CDC baked in). The reference implementation of the book's patterns.
• Axon Framework — Java; CQRS + Event Sourcing with first-class sagas (@Saga, @SagaEventHandler, SagaLifecycle.end()). Natural when you're already event-sourced.
• MassTransit — .NET; both a saga state machine (Automatonymous) and the routing-slip pattern (Courier).
• NServiceBus — .NET; mature sagas with timeouts and persistence.
• Seata (originally Alibaba; now Apache Seata) — explicitly ships multiple modes, a neat way to remember the taxonomy: AT (Automatic Transaction — auto-generates reverse SQL, a 2PC-like compensation with a global lock), TCC (you write Try/Confirm/Cancel), Saga (long-running, state-machine driven, compensation-based), and XA (true 2PC over XA resources). One product, four points on the consistency/isolation spectrum. (XA was added in v1.2/2020, so older write-ups list only three modes — a handy version-awareness note.)

Transport under all of this: Kafka, RabbitMQ, or SQS for events/commands/replies, with Debezium commonly providing the outbox CDC (§10.1).

14. Problems & how to solve them (Problem → Root cause → Fix)#

The "what goes wrong with sagas" question. Each is a place real systems break.

14.1 Non-compensatable action already happened#

Symptom: a step you can't undo (email sent, money captured non-refundably, physical goods shipped) ran, then a later step failed and you "need to roll back." Cause: irreversible work placed before the point of no return. Fix: the pivot (§4). Put irreversible/must-happen work after the pivot as retriable steps, and reversible/risky work before it. If something truly can't be reversed and can't be deferred, redesign the flow so it's the pivot itself. Never leave an irreversible step in the compensatable region.

14.2 Compensation fails#

Symptom: the undo itself errors; system left half-done. Cause: downstream unavailable, or a non-idempotent compensation double-applied. Fix: idempotent compensations + retry with backoff/jitter; bounded retries then park in COMPENSATION_FAILED and alert a human (§10.3). Design compensations to be local and cheap so they rarely fail.

14.3 Dirty read of in-flight saga state#

Symptom: another process acts on data a saga later compensated (counted revenue for an order that got cancelled). Cause: missing isolation (§9); intermediate committed state read as if final. Fix: semantic lock — expose a *_PENDING state and make readers honor it (skip/wait/alternate path); clear it on completion or compensation. Consider pessimistic view (reorder steps) to shrink the window.

14.4 Lost update / oversell under concurrency#

Symptom: two sagas both "succeed" on a resource that only one should get (last unit sold twice). Cause: concurrent read-modify-write without isolation. Fix: optimistic version check (reread value) to detect the conflict and abort/retry; commutative updates (reserve via decrement, not absolute set); a semantic lock on the contended resource. For inventory specifically, reserve-then-confirm (TCC-style) so the unit is held, not just counted.

14.5 Dual-write between DB and broker in a step#

Symptom: a step changed its DB but no event was published (saga stalls), or published without the DB change. Cause: save() + send() as two non-atomic calls (§10.1). Fix: Transactional Outbox (+ Debezium CDC). Emit from the outbox, never directly alongside the DB write. Consumers dedup because outbox delivery is at-least-once.

14.6 Duplicate processing#

Symptom: a step or compensation runs twice (double refund, double reserve). Cause: at-least-once delivery; retries; outbox re-publish. Fix: idempotent handlers keyed on message/business ID (unique constraint, upsert, seen-set) (§10.2). Assume duplicates; make them harmless.

14.7 Stuck / hung saga#

Symptom: a saga never finishes; semantic locks held; other work blocked. Cause: a participant never replied (crash, lost message, poison pill); no timeout. Fix: per-step timeouts with a retry-or-compensate decision; watchdog for choreographed steps; metrics/alerts on saga age and state; a way to manually resolve (§10.4).

14.8 Out-of-order events break a choreographed saga#

Symptom: a later event is processed before an earlier one; state machine confused. Cause: events for one saga spread across partitions/queues without ordering, or concurrent producers. Fix: key messages by sagaId/entity so they share a partition (per-key ordering); make handlers tolerate out-of-order with version/sequence checks; consider a version file (§9) to reorder/reconcile.

14.9 Cyclic event chains (choreography)#

Symptom: services trigger each other in a loop; runaway events. Cause: emergent choreography graph with a cycle. Fix: map the event graph; break cycles; if the flow is this tangled, it's a signal to switch to orchestration for explicit control.

14.10 "God orchestrator"#

Symptom: business rules pile up in the orchestrator; services become anemic CRUD shells. Cause: leaking domain logic into the coordinator. Fix: keep the orchestrator about sequencing + compensation only; push domain decisions into the owning services (they reply with outcomes, the orchestrator just routes).

14.11 Assuming the saga is isolated / ACID#

Symptom: design assumes no one sees intermediate state; anomalies in prod. Cause: forgetting the missing "I". Fix: state ACD-not-ACID up front; apply §9 countermeasures deliberately. This awareness is the senior signal.

15. Interview framing — how to sound senior#

• Define it in one breath, including the trade: "A sequence of local transactions with compensating transactions; you trade isolation and instant rollback for availability and loose coupling; it's eventually consistent." Saying the trade-off unprompted is the tell.
• Say ACD-not-ACID and mean it. Immediately follow with "so I design for dirty reads and lost updates with semantic locks and version checks." This one move separates people who've read about sagas from people who've built them.
• Reach for the pivot when asked about ordering or irreversible steps. Compensatable → pivot → retriable is high-signal and most candidates don't know it.
• Pick coordination by complexity, not dogma: choreography for simple/stable/few-step; orchestration for complex/observable/many-compensation; hybrid across bounded contexts. Never say "orchestration is always better" or vice versa.
• Volunteer the outbox the instant a step "updates the DB and publishes an event." Dual-write is the correctness hole in every saga step.
• Insist on idempotency for handlers and compensations, because delivery is at-least-once.
• Contrast with 2PC deliberately: "I avoid 2PC because it blocks holding locks, the coordinator can leave participants in-doubt, it's CP, and half my stack (Kafka, that NoSQL store) has no XA." Then: "and the best distributed transaction is the one I avoid with better bounded contexts."
• Know TCC as the isolation upgrade ("reservation-based saga — Try holds resources invisibly, so no dirty reads, at the cost of three methods per participant") and routing slip as the itinerary-in-the-message hybrid.
• Name real tools (Temporal / Camunda-Zeebe / Step Functions / Eventuate / Axon / Seata's four modes) and say why — durable execution, BPMN visibility, serverless, etc.
• Talk operations: timeouts, stuck-saga detection, saga-age metrics, COMPENSATION_FAILED alerts. Sagas are an operational commitment, not just a design.
• Cite the lineage: Garcia-Molina & Salem 1987 (origin); Chris Richardson's Microservices Patterns (the modern taxonomy — coordination types, pivot, countermeasures). Lands well.

One-paragraph "design a distributed transaction" template#

"First I'd try to avoid it — can one bounded context own this invariant and use a single local transaction? If it genuinely spans services, I'd use a saga: local transactions with semantic compensations, eventually consistent. For a multi-step, branching, must-be-observable business process I'd orchestrate it (Temporal or Camunda) so state, timeouts, retries, and compensation live in one place; for a simple 2–3 step flow I'd choreograph with events. Each step uses the transactional outbox (Debezium CDC) so the DB write and the emitted event are atomic, and every handler and compensation is idempotent because delivery is at-least-once. I order steps as compensatable → pivot → retriable so irreversible work only runs once we're committed to completion. Since a saga has no isolation, I add semantic locks (*_PENDING states) for dirty reads and optimistic version checks / commutative updates for lost updates. I run per-step timeouts, detect stuck sagas, and alert on failed compensations. If I needed stronger isolation without 2PC, I'd move the contended steps to TCC reservations."

16. Decision cheat-sheet#

17. Self-test#

If you can answer these cold, you're ready:

1. Why is a saga ACD and not ACID, and which single missing property causes most of its complexity?
2. Give three concrete reasons microservices avoid 2PC, and one situation where 2PC is still the right call.
3. What is a compensating transaction, and why is "semantic, not physical rollback" more than pedantry? Give an example where the end state differs from the start state.
4. Classify the steps of an order saga into compensatable / pivot / retriable and explain why the ordering minimizes risk.
5. Contrast choreography and orchestration on visibility, coupling, and failure handling — and name a case for each.
6. Name the three saga anomalies and match each to a countermeasure (semantic lock, commutative update, pessimistic view, reread value, version file, by value).
7. Why does every saga step need the transactional outbox, and why do consumers still need to dedup after it?
8. Why must compensations be idempotent, and what do you do when a compensation itself keeps failing?
9. How does TCC achieve better isolation than a plain saga, and what does it cost you?
10. Where does a stuck saga come from, and what machinery detects and resolves it?
11. What's the difference between backward and forward recovery, and how does the pivot decide which one you're in?
12. Sketch how you'd implement the same order saga in Temporal vs Kafka choreography, and when you'd choose each.

Lineage: the saga concept is from Garcia-Molina & Salem, "Sagas" (1987); the modern microservices taxonomy — choreography vs orchestration, the compensatable/pivot/retriable classification, and the isolation countermeasures — follows Chris Richardson's Microservices Patterns. Durable-execution framing follows Temporal/Cadence; BPMN orchestration follows Camunda/Zeebe.