Schema Evolution in EDA & Event Sourcing

How to change the shape of your events over years of production without a synchronized deploy — the discipline behind Schema Registry compatibility modes and Event-Sourcing upcasters, and the Staff+ interview questions that probe whether you actually understand it.

See also: kafka-eda-study-guide.md, eda-vs-event-sourcing-study-guide.md, event-sourcing-study-guide.md (§9 covers ES upcasting — this guide is the dedicated, wider treatment), event-broker-vs-message-bus-study-guide.md, inbox-outbox-pattern.md, API-Design-Best-Practices.md (versioning a REST contract is the same problem one layer up).

The mental model#

Every serialized event is a one-way message from a writer in the past to a reader in the future who cannot ask for clarification. Schema evolution is the discipline of keeping writers and readers that never meet and never upgrade at the same instant mutually intelligible. Strip away the tooling and it reduces to a single question:

Which side is allowed to be older — the reader or the writer — and for how long?

That one question forks into the two regimes you'll be asked about, and the whole guide hangs off this fork:

• EDA (transport). Reader and writer are different services, skewed by the duration of a rolling deploy — minutes to days. The skew is bounded: you eventually finish deploying and the old version is gone. A Schema Registry sits at the boundary and enforces a compatibility rule so an incompatible schema can't even be published. You negotiate compatibility and the gate holds the line.

• Event Sourcing (storage). The "writer" is your own code from years ago; the reader is your code now, folding a five-year-old event to reconstruct state. The log is immutable and infinite — you can neither rewrite the past nor reject it. The skew is unbounded and there is no gate. So you don't negotiate; you translate on read — an upcaster rewrites the old shape into the current one in memory, forever.

Same discipline, two regimes: bounded skew with enforcement (EDA) versus unbounded skew with translation (ES). Almost every mistake in this area comes from applying one regime's instincts to the other — e.g. "the registry will catch it" (there is no registry over your event store) or "we'll just run a migration" (you cannot migrate an immutable, infinite log in one shot). Hold the fork in your head and the rest is derivation.

A note on layers, because interviewers conflate them on purpose: EDA and ES are orthogonal (see eda-vs-event-sourcing-study-guide.md). A system can be neither, either, or both. When it's both, it has two independent schema-evolution problems — the private domain-event log inside the service, and the public integration-event contract on the wire — and they are governed differently. Keeping those two straight is a senior signal all by itself; §7 is entirely about their correlation.

1. The fundamental problem: writer/reader skew#

Nothing here makes sense until you internalise why you can't just "change the class and redeploy."

You never control the order in which producers and consumers reach the new version. Three facts guarantee mixed-version traffic:

1. Rolling deploys. During a deploy, v1 and v2 instances of the same service run simultaneously. A v2 producer's message may be read by a v1 consumer still on the old pods, and vice versa.
2. At-least-once delivery + retention. A broker replays. A message written by v1 last Tuesday is consumed by v3 today. In Event Sourcing this is extreme: an event written in 2021 is folded by 2026 code on every rehydration.
3. Independent teams. In EDA the consumer is another team on their own release train. You do not get to coordinate their deploy with yours — that coupling is the exact thing EDA exists to avoid.

So at any moment, data written under schema A is being read by code expecting schema B, for many (A, B) pairs at once. Schema evolution is the set of rules that keep every actually-coexisting (writer, reader) pair intelligible.

The vocabulary you must anchor (this is the #1 source of confusion)#

Avro makes the core idea explicit and it's worth adopting its framing everywhere: there is a writer schema (the schema the bytes were written with) and a reader schema (the schema the reading code expects). Reading is schema resolution between the two.

Define compatibility from the reader's point of view:

• Backward compatible = a new reader can read old data. (The new schema is "backward-compatible with" what came before.) You upgrade consumers first; once they understand both old and new, producers can start emitting new.
• Forward compatible = an old reader can read new data. (Old code tolerates data written by a newer schema.) You upgrade producers first; consumers still on the old schema keep working.
• Full = both directions hold with the adjacent version.

Interview trap: people say "backward compatible" to mean "old clients still work," which is actually forward compatibility in registry terms. Pin the definition to who is the reader and you'll never flip it. The mnemonic: BACKWARD = new code reaching BACK to read old data.

2. Serialization formats — the substrate of evolution#

Your format choice is a schema-evolution decision, made before you write a line of business code. The question a format answers is: when the reader's schema and the bytes disagree, what happens?

The three that matter in JVM/Spring interviews are JSON (via Jackson), Avro, and Protobuf. What you need to be able to say about each:

JSON + Jackson — schema-on-read. There is no schema on the wire and no enforcement. Your only evolution tool is the tolerant reader: ignore unknown fields, default missing ones. Cheap, human-readable, ubiquitous — and completely ungoverned, which is why it's the default inside an event store (you own both ends) and a liability across teams (nobody stops a breaking change). Fields are matched by name, so a rename is always a breaking change unless you read both names.

Avro — writer schema + reader schema resolution. Compact binary, rich types, and it demands the writer schema be available at read time to resolve against the reader schema. Shipping the full schema in every message would dwarf the payload — hence the registry, which lets each message carry a tiny schema ID instead. Fields match by name; evolution leans entirely on defaults (a field with a default can be added or removed compatibly; one without cannot). This is the format the Confluent stack is built around, and the one most schema-evolution interview questions implicitly assume.

Protobuf — tag numbers are the contract. Fields are identified by an integer tag, never the name, so you can freely rename fields in the IDL and the wire stays stable. Unknown fields are preserved (not dropped) through a read-modify-write, which is great for forward compatibility. proto3 has no required, every scalar has a zero default, and removed fields must be reserved so their numbers are never reused. Protobuf is the most evolution-friendly format almost by construction — the trade-off is that "the number means the field" is a discipline humans get wrong.

The golden rules that hold for every format#

Independent of the tool, a senior candidate states these as invariants:

• Never reuse a name (Avro/JSON) or a tag number (Protobuf). Reuse silently reinterprets old bytes as a new field — the nastiest class of data-corruption bug because nothing errors.
• Never change a field's type incompatibly. Widening is sometimes safe (Avro int→long, float→double); narrowing and unrelated changes are not. When in doubt, add a new field.
• Additive-with-a-default is the safe change. Adding an optional field with a sensible default is the one move that's compatible in both directions almost everywhere. It's the backbone of expand/contract (§8).
• Never change the meaning of an existing field or event. amount in cents must mean cents forever. Semantic drift passes every compatibility checker and every test, and quietly poisons five years of history. This one bites hardest in Event Sourcing (§6).

3. Compatibility modes — the core registry concept#

This is the single most-tested topic in this area. Confluent Schema Registry enforces one of these compatibility modes per subject; the mode decides which schema changes are legal and — the part juniors miss — who has to deploy first.

Derive it, don't memorise it#

You should be able to reconstruct the "permits" column from first principles in the interview — that's the difference between recall and understanding. Reason about a single change and both directions (Avro semantics):

• Add a field with a default. New reader on old data → uses the default → OK (backward). Old reader on new data → ignores the unknown field → OK (forward). Compatible both ways ⇒ legal under FULL. This is why "add optional field" is the universally safe move.
• Add a field without a default. New reader on old data → field missing, no default → fails backward. Old reader on new data → ignores it → OK forward. So it's FORWARD-only — you must upgrade producers first.
• Delete a field with a default. New reader (field gone) on old data → ignores it → OK backward. Old reader (still expects field) on new data lacking it → falls back to its default → OK forward. Both ways ⇒ FULL.
• Delete a field without a default. New reader ignores it on old data → OK backward. Old reader needs it, new data omits it, no default → fails forward. So BACKWARD-only.

Notice the pleasing symmetry: adding is forward-safe, deleting is backward-safe, and doing either with a default is both. That single sentence lets you answer any "is this change compatible?" question live.

The insight that sounds senior: mode ⇒ deployment order#

Compatibility mode is not academic; it dictates your rollout runbook:

• BACKWARD (the default): new consumers can read both old and new data, so roll out consumers first, then producers. This matches how you usually want to deploy — get every reader ready before any new data appears. It's the default precisely because "consumers ready before new events" is the safe, common case, and Event Sourcing requires this shape (a new fold must read all historical events).
• FORWARD: old consumers can read new data, so you can roll out producers first and let consumers catch up. Good when producers are on a faster cadence, or when you must start emitting a new field immediately and consumers will adopt it later.
• FULL: either order works — the strongest guarantee and the least freedom (optional add/remove only). Reserve it for contracts where you genuinely can't coordinate deploy order across teams.

Transitive vs non-transitive — the trap in the trap#

Non-transitive modes check only against the latest registered schema. That admits a slow-motion break: v1→v2 legal, v2→v3 legal, yet v1→v3 incompatible. If a consumer or a retained message is still on v1 when v3 data arrives, it breaks even though every step "passed." *_TRANSITIVE checks against the entire version history, closing that hole — at the cost of never being able to shed an old constraint. For an event stream with long retention (and always for an event store, where v1 events live forever), transitive is the honest choice. Volunteering this distinction unprompted is a strong signal.

When NONE is the right answer#

NONE isn't just "we turned safety off." It's correct when you're deliberately making a breaking change via a new subject/topic (§8) and will manage the migration yourself, or in a dev sandbox. Saying "I'd set NONE on this new v2 topic because the break is intentional and handled by dual-publishing, not by pretending it's compatible" is a mature answer; silently running NONE in prod is a red flag.

4. Schema Registry — the EDA correlation, in depth#

The question "what's the correlation with Schema Registry?" has a precise answer: the registry is the enforcement point and single source of truth for the transport contract in EDA. It's where the abstract "compatibility rule" from §3 becomes a wall that a bad deploy hits in CI instead of a corruption your consumers hit at 2 a.m. Here's what a Staff+ candidate knows about it.

What it actually is#

A standalone HTTP service holding versioned, immutable schemas keyed by subject. Producers/consumers talk to it through their serializers, not directly. It does exactly three things: store schemas (assigning each a global ID and a per-subject version), serve them by ID, and enforce the subject's compatibility mode at registration time — rejecting an incompatible schema with an HTTP 409 before a single bad message is produced. It stores schemas, not data, and it validates structure, not business rules (it will happily accept a semantically wrong-but-structurally-compatible schema — semantic drift is still your job).

The wire format — why messages are tiny and what it costs you#

A Confluent-serialized Avro message is not self-describing JSON. The bytes are:

(Protobuf adds a message-index array after the ID; JSON Schema is analogous.) Consequences to state out loud:

• The message carries an ID, not a schema — a few bytes of overhead instead of kilobytes. The deserializer reads the ID, fetches that exact writer schema from the registry (cached after first use), and resolves it against the consumer's reader schema.
• The registry is on the serialization path. It must be reachable when a new schema ID is first seen. Client-side caching means it's not hit per-message, but a registry outage can stall first-time (de)serialization — so registries run HA, and clients cache aggressively. This is the operational cost of the compact wire format.
• Reader schema ≠ writer schema, by design. The consumer resolves the writer schema it fetched against its own compiled reader schema. That resolution is schema evolution happening at runtime — defaults filled, unknown fields dropped, per §3.

Subjects & naming strategies (the multi-event-type question)#

A subject is the unit that has a version history and a compatibility mode. Which subject a message maps to is set by the subject naming strategy — and this is a genuine design decision, not a default to ignore:

Why it matters and where it ties to Event Sourcing: if you need several event types on one topic in strict order — e.g. OrderPlaced, OrderPaid, OrderShipped on one partition keyed by order id, so a consumer sees them in sequence — TopicNameStrategy breaks, because it forces one -value schema per topic. Switch to RecordNameStrategy/TopicRecordNameStrategy (plus auto.register.schemas/use.latest.version tuning) and each event type keeps its own subject and its own compatibility history while sharing the partition. This is exactly the shape you get when you publish an event-sourced aggregate's stream to Kafka, so the question "how do you put multiple event types on one ordered topic?" is really a subject-naming question.

Registration & governance — shift the gate left#

Two configs decide when the compatibility gate fires:

• auto.register.schemas — in dev, true: the producer registers its schema on first send. In production, set it false. You do not want a rogue producer silently registering a new schema at runtime; you want schemas registered deliberately, and the compatibility check to run in CI, before merge.
• use.latest.version=true (with auto-register off) tells the serializer to use the latest registered schema rather than the one compiled in, which supports certain controlled-rollout patterns.

The governance win is shift-left: wire the kafka-schema-registry-maven-plugin (goal test-compatibility) or its Gradle equivalent into CI so a pull request that would break orders-value fails the build, with the registry as the source of truth it checks against. The registry stops being a runtime dependency you hope holds and becomes a design-time contract test. That reframing — "the registry lets me catch a breaking change in code review, not in production" — is the senior articulation of why it exists.

Schema references#

Avro and Protobuf support references — one schema embedding another by name/version (a shared Address, Money, or a union of event types for a multi-type topic). References let you version shared value types once and reuse them, rather than copy-pasting a nested record into every schema. Mentioning them signals you've run a real registry at scale, not just the quickstart.

5. EDA schema evolution in practice (code)#

Now make it concrete. An OrderPlaced integration event, Avro, on Kafka, governed by Schema Registry in BACKWARD mode.

v1 — the initial schema#

src/main/avro/OrderPlaced.avsc:

The Avro Gradle/Maven plugin generates a SpecificRecord class OrderPlaced from this. Producers and consumers depend on the generated type.

The safe change: add an optional field (BACKWARD/FULL compatible)#

Marketing wants a coupon code. The safe move is additive with a default — legal in BACKWARD (and FULL). OrderPlaced.avsc v2:

["null","string"] with "default": null is the Avro idiom for "optional." A v1 consumer reading a v2 message drops the unknown couponCode (forward); a v2 consumer reading a v1 message fills null from the default (backward). Nobody has to coordinate — this is why 90% of real changes should be shaped this way.

Spring Boot producer & consumer#

Configuration (application.yml) — the serdes and registry wiring are the point:

Kotlin

Java

Guard it in CI, not at runtime#

The producer doesn't self-register in prod (auto.register.schemas: false); the compatibility gate runs in the build. Maven:

A PR that deletes currency (a required field) now fails the build with an incompatibility error, against the real registry, before merge. That's the whole value proposition in one config block.

The breaking change — when additive isn't enough#

Some changes can't be made compatible: splitting amountCents into netCents + taxCents, changing orderId from string to a structured type, or removing a field consumers still require. You do not force it onto the same subject. Options, in rough order of preference:

1. Reframe it as additive (expand/contract, §8). Add netCents/taxCents alongside amountCents, populate all three, let consumers migrate, remove amountCents only once nothing reads it. Most "breaking" changes dissolve into a sequence of compatible ones with an overlap window. This is the answer interviewers want first.
2. New topic / new subject (orders.v2) with the incompatible schema. Producers dual-publish to orders.v1 and orders.v2 during the transition; consumers move at their own pace; retire v1 when idle. This is the honest home for a truly incompatible redesign, and where a deliberate NONE on the new subject is reasonable.
3. A versioned event type (OrderPlacedV2 as a distinct record) on the same or a new topic, with consumers handling both — closer to the ES approach in §6.

Protobuf — same story, tag-numbered#

If the shop is on Protobuf (KafkaProtobufSerializer/KafkaProtobufDeserializer), the evolution rules live in the field numbers:

Adding coupon_code = 6 is safe because old readers skip unknown tags and preserve them on re-serialize (proto3 dropped unknown-field retention briefly but restored it in 3.5). Removing a field means reserving its number and name so a future change can't recycle tag 5 and silently reinterpret old bytes. proto3 gives every scalar a zero-value default, so by default "missing" and "zero" are indistinguishable on the wire — you can't tell "0 dollars" from "unset." When you need that distinction, proto3's optional keyword (explicit field presence, stable since 3.15) restores it without falling back to wrapper types.

6. Event Sourcing schema evolution — the same discipline, played on hard mode#

Everything above assumed a bounded window (a deploy) and a gate (the registry). Event Sourcing removes both, and that changes the tactics even though the theory is identical. This section complements event-sourcing-study-guide.md §9 — read that for the base mechanics; here's the wider decision framework.

Why it's genuinely harder — three structural differences#

1. The past is immutable and already written. In EDA a bad schema is rejected before it produces data. In ES the events already exist; there is no "reject" — only "cope." You cannot un-write 2021.
2. There is no gate. No registry sits between your 2021 code and your 2026 fold. The compatibility requirement still exists ("today's code must read every event ever written"), but nothing enforces it for you — you enforce it in code and tests, or you find out at replay time.
3. The window is infinite. EDA compatibility is effectively "the latest version, maybe a few back." ES compatibility is every version that has ever existed, forever — which is exactly BACKWARD_TRANSITIVE taken to its limit. A new fold is a new reader that must read all historical writers. This is why ES is intrinsically a "consumers-ready-first" world and BACKWARD is its native mode.

Net: you can't negotiate with the past, so you translate it. The registry's job (block incompatible writes) has no ES analogue; the consumer-side job (resolve old data to the current shape) becomes your whole life, and you do it explicitly with upcasters.

The evolution ladder — reach for the lowest rung that works#

Handle a change with the least powerful tool that suffices; escalate only when forced:

1. Tolerant reader — additive changes. Ignore unknown fields, default missing ones. With Jackson: @JsonIgnoreProperties(ignoreUnknown = true) and nullable/defaulted properties. Handles the majority of real changes for free, same as EDA's "add optional field."
2. Upcaster — structural changes. A pure function that transforms an old event version into the current shape at read time. The stored bytes never change; you translate on the way out of the store. Rename a field, change units, split/merge — all upcasters.
3. Versioned event type — when a change is too big to upcast cleanly. Keep MoneyWithdrawnV1 and MoneyWithdrawnV2 as distinct types and handle both in the fold. Useful when v1 and v2 carry genuinely different information.
4. Copy-and-replace stream migration — the last resort (Greg Young's "Copy and Replace"). Read the old stream, write a brand-new stream (new stream id / new store) with transformed events, cut over. The only tool that changes stored history, and you use it only when upcasting would be absurdly complex or you're consolidating years of accreted versions. Expensive, risky, done rarely and deliberately.

The senior instinct: stay on rungs 1–2. Every rung-4 migration is a small data-migration project with its own rollback plan.

Greg Young's taxonomy — the change types and the cardinal rule#

Greg Young's Versioning in an Event Sourced System is the canonical reference; interviewers who know ES deeply expect its vocabulary. The change types, mapped to the ladder:

• New event type → just start emitting it; folds ignore what they don't handle. Trivial.
• Add a field → tolerant reader + default. Rung 1.
• Rename a field / change a type / change units (cents→millis) → upcaster. Rung 2.
• Split one event into two, or merge two into one → upcaster that emits/absorbs, or a versioned type. Rung 2–3.
• Change of meaning → forbidden. You never redefine what an existing event means. Instead, emit a new event type going forward and leave the old one meaning exactly what it always did.

The cardinal rule underneath all of it: an event is a historical fact; facts don't change. If the business concept changed, that's a new fact with a new type — not an edit to an old one. This is the ES restatement of §2's "never change meaning," and it's the line that separates people who've run ES from people who've read about it.

Upcasting in code#

Version the type name so you know which shape you're holding (event_type = 'MoneyWithdrawn.v1'), then walk a chain of small transforms v1 → v2 → v3 at read time.

Kotlin

Java

Domain code only ever sees MoneyWithdrawn.v2 — the fold never learns v1 existed. Keep the upcaster forever (or until a rung-4 migration retires every v1 event). This is exactly how Axon Framework's upcaster chain works (SingleEventUpcaster / EventUpcasterChain); EventStoreDB leaves it to you in application code. Framework APIs are version-dependent — present the shape, not memorised annotations.

Two schemas, not one — and the snapshot gotcha#

An event-sourced service actually has two evolving schemas, and conflating them is a classic mistake:

• The event schema — governed by upcasting, forever (above).
• The read-model / projection schema — a derived store you can throw away and rebuild from events. When it changes, you don't upcast; you replay into a fresh projection (blue/green, then cut over). This is a superpower EDA-only systems lack: your read side has no legacy-data problem because it isn't the source of truth.

And snapshots are versioned by the state schema, not the event schema. If the aggregate's serialized shape changes, old snapshots may not deserialize — the safe rule is discard any snapshot you can't read and rebuild from events. A snapshot must never become un-reproducible; it's a cache, not a record.

GDPR — the immutability tax#

"Delete my data" versus an immutable, infinite log is the other ES-specific evolution pressure. You can't rewrite events, so: crypto-shredding (encrypt PII per subject, delete the key to render events unreadable) or keep PII out of events entirely (store a reference to a mutable PII table). Interviewers pair this with schema evolution because both stem from the same root — the log is forever — and both are the price of the audit trail.

7. Schema Registry ↔ Event Sourcing — how they actually relate#

This is the question that separates people who've read two blog posts from people who've designed the boundary. The crisp framing: a registry governs a transport contract; upcasting governs a stored history. They solve overlapping-looking problems in non-interchangeable ways.

Can you use a registry for your event store?#

Yes, partially — and knowing exactly which part is the senior nuance. You can absolutely encode stored events with Avro/Protobuf and even register those schemas so codegen and tooling are shared with your Kafka stack. What a registry cannot do for an event store is be the safety mechanism: its compatibility check answers "can a reader read a writer?" for the versions it knows about, but ES needs the strictly stronger "can today's fold reconstruct state from every version ever written?" — and it must hold for data that already exists and cannot be rejected. Set the subject to FULL_TRANSITIVE and you get closer, but you still need upcasting to actually transform old bytes into the current shape at read time; the registry validates, it does not translate. So: registry for encoding and shared tooling, upcasters for the guarantee. "The registry checks; the upcaster copes" is a one-liner that lands.

The boundary in a combined system — two contracts, two disciplines#

The standard senior design (from eda-vs-event-sourcing-study-guide.md §5) makes the relationship physical: event-source internally, publish integration events externally through an outbox. That boundary is also the schema-governance boundary:

Two separate schemas evolve on the two sides of that outbox, and you must not let them be the same object:

• Inside: private, fine-grained domain events (ItemAddedToCart, DiscountApplied), evolved by upcasting, governed by nobody but you, kept forever.
• Outside: coarse, public integration events (OrderPlaced), evolved under a registry with a compatibility mode, versioned like a REST API, consumed by teams you don't control.

The anti-pattern interviewers fish for: publishing your raw domain events straight onto the registry-governed topic. Now every internal refactor is a breaking change to other teams, and the registry either blocks your refactor or (worse, on NONE) lets it through and breaks them. Translate domain → integration at the boundary (in the outbox write) so the two evolution problems stay independent. That translation is the correlation between the two worlds: the registry protects the contract you chose to expose, upcasting protects the history you're obligated to keep.

8. The universal playbook: expand → migrate → contract#

One recipe makes any breaking change safe in both worlds, and it's the money answer to "how do you change this without downtime or coordinated deploys?" It's called parallel change (a.k.a. expand/contract). The insight: you can never flip atomically across an uncoordinated fleet, so you always pass through an overlap window where old and new coexist — engineer that window instead of pretending it away.

Three phases:

1. Expand. Introduce the new shape alongside the old. Producers write both (or the new field defaulted); the store/topic now carries old and new forms. Nothing reads the new form yet, so this phase is compatible by construction.
2. Migrate. Move readers to the new shape. In EDA: deploy consumers that understand the new field/type (BACKWARD mode means they read both). In ES: add the upcaster / new fold so historical and new events both resolve to the current shape. Backfill or upcast the old data. Producers can now stop writing the old form.
3. Contract. Once nothing reads or writes the old shape, remove it. In EDA: delete the old field/topic and let the registry confirm it's now safe. In ES: retire the old event class from the fold — but keep the upcaster (the old bytes still exist and still need translating; only a rung-4 migration ever removes them).

Mapping the same recipe onto the tools you've already seen:

Two things to say out loud that signal maturity: the overlap window is mandatory, not a smell — its existence is why zero-downtime change is possible; and deployment order falls out of the compatibility mode (§3), so "which do I deploy first?" is answered by "what mode is the subject in?", not by guesswork. Feature flags, canaries, and dual-writes are just tactics for controlling the width of that window.

9. Problems & how to solve them#

Problem: a consumer started NPE-ing / throwing deserialization errors right after a producer deploy. Root cause: producer added a field with no default under BACKWARD mode, or the schema wasn't actually registered/checked — new data has a field old readers can't resolve. Fix: make the field optional-with-default (additive); ensure the compatibility gate runs in CI (test-compatibility), not at runtime; if the change is truly breaking, use a new subject and dual-publish (§8).

Problem: a change "passed" compatibility on every step but an old consumer still broke. Root cause: non-transitive mode only checked against the latest schema; v1→v3 was never validated, and a v1 reader (or retained v1 message) is still live. Fix: switch the subject to a *_TRANSITIVE mode so every historical version is checked; for long-retention topics and event stores, transitive is the default you want.

Problem: replaying an event stream crashes after a refactor. Root cause: a domain event's shape changed but the store holds the old shape forever; the current fold can't read it. Fix: tolerant reader for additive changes; upcaster for structural ones (translate old→current on read); never rewrite stored events (§6). Add a replay test over a corpus of historical event fixtures so this fails in CI.

Problem: someone "fixed a bug" by changing what an existing event means (e.g. amount now excludes tax). Root cause: semantic change to an immutable fact — passes every structural compatibility check, silently poisons all history before and after the change. Fix: revert; introduce a new event type for the new meaning and leave the old one meaning what it always did. Meaning is never editable (§2, §6). Code review must treat "change the meaning of a field" as a breaking change.

Problem: every internal refactor breaks other teams' consumers. Root cause: raw domain events are published directly onto the public, registry-governed topic — internal shape is the external contract. Fix: translate domain → integration events at the outbox boundary; version the integration event like an API; keep the two schemas independent (§7).

Problem: the schema registry went down and (de)serialization stalled. Root cause: first-time schema-ID lookups are on the serialization path; a cold cache plus a registry outage blocks progress. Fix: run the registry HA; rely on client-side schema caching; pre-warm/pin known schema IDs; treat the registry as a tier-1 dependency. It's rarely hit per-message, but it is on the critical path for new IDs.

Problem: multiple event types must share one ordered topic, but the topic only accepts one schema. Root cause: default TopicNameStrategy binds one -value schema per topic. Fix: switch to RecordNameStrategy / TopicRecordNameStrategy so each event type is its own subject with its own compatibility history while sharing the partition; key by aggregate id for ordering (§4).

Problem: old snapshots fail to deserialize after an aggregate-state change; service won't start. Root cause: snapshots are versioned by the state schema, which drifted. Fix: discard un-readable snapshots and rebuild from events (a snapshot is a cache, never a record); version snapshot payloads so you can detect drift (§6).

Problem: GDPR erasure request against an immutable event log. Root cause: immutability is the feature and the obstacle. Fix: crypto-shredding (per-subject key, delete key on erasure) or keep PII out of events and reference a mutable store (§6).

10. Interview framing — how to sound senior#

• Open with the fork, not the tooling. "Schema evolution is keeping past writers and future readers intelligible without a synchronized deploy. In EDA the skew is bounded and a registry enforces compatibility; in event sourcing the skew is infinite and there's no gate, so you upcast on read. Same problem, two regimes." That two-sentence frame organises everything that follows.
• Nail backward vs forward from the reader's side, and connect it to deploy order. "BACKWARD = new code reads old data → upgrade consumers first. FORWARD = old code reads new data → upgrade producers first. FULL = either order. The compatibility mode is the rollout runbook." Deriving the deploy-order consequence unprompted is the strongest single signal here.
• Derive the change-safety rules live: adding is forward-safe, deleting is backward-safe, either-with-a-default is both. Don't recite a table — reason it out; it shows you understand why.
• Volunteer the registry's role and its limits. It's the single source of truth and a CI gate (auto.register.schemas=false, test-compatibility shift-left) — and it checks structure, not meaning, and governs transport, not your stored history.
• Explain the ES difference precisely: no gate, infinite window, can't reject or rewrite the past ⇒ upcasting; effectively BACKWARD_TRANSITIVE forever; never change an event's meaning — new fact, new type.
• Answer "how do you make a breaking change?" with expand/contract, not "we'll coordinate a deploy." The overlap window is mandatory; engineer it.
• Draw the boundary in a combined system: private domain events (upcasted) inside, public integration events (registry-governed) outside, translated at the outbox — two independent schema-evolution problems. This is the direct answer to "how do the registry and event sourcing relate?"
• Wrong answers to avoid: "the registry will catch it" (there's no registry over an event store); "we'll just migrate the events" (you can't migrate an immutable, infinite log in one shot — you upcast); "backward compatible means old clients still work" (that's forward); "just version the whole event v1→v2 for every change" (most changes should be additive; reserve new types for real breaks); "Kafka retention = event store, so same evolution story" (different guarantees — see eda-vs-event-sourcing-study-guide.md).

11. Cheat sheets#

Compatibility modes#

Change-safety matrix (Avro semantics)#

EDA vs ES evolution#

The evolution ladder (ES) — lowest rung that works#

1. Tolerant reader → 2. Upcaster (read-time transform) → 3. Versioned event type → 4. Copy-and-replace migration (last resort).

12. Self-test#

1. Define backward vs forward compatibility from the reader's perspective, and state which side you deploy first for each. Why is BACKWARD the registry default?
2. A subject is in non-transitive BACKWARD mode. v1→v2 and v2→v3 both registered fine, yet a consumer broke. What happened, and which mode prevents it?
3. Derive from first principles: is "add a field without a default" backward-compatible, forward-compatible, both, or neither? Now do "delete a field without a default."
4. Walk the Confluent Avro wire format byte-by-byte. Why is a registry outage sometimes a problem despite client caching?
5. You must carry OrderPlaced, OrderPaid, OrderShipped on one ordered topic. What breaks with the default subject naming strategy, and what do you change?
6. In event sourcing there's no schema registry over the store. What enforces that today's code can read a 2021 event, and what happens if nothing does?
7. A teammate wants to "fix" an event by changing what an existing field means. Why is this worse than a structural break, and what should they do instead?
8. Explain the exact relationship between Schema Registry and upcasting. Can a registry replace upcasting for an event store? Why or why not?
9. Make a breaking change (amountCents → netCents + taxCents) with zero downtime and no coordinated deploy. Walk the three phases and say where the old form finally dies — in EDA and in ES.
10. In a service that is event-sourced internally and publishes integration events externally, how many schemas evolve, governed how, and what's the anti-pattern that collapses them into one?

(Answers derive from §1–§8; if any feels shaky, that section is where to look.)

Sources & lineage#

• Confluent Schema Registry documentation — compatibility types (BACKWARD/FORWARD/FULL + transitive/NONE), subject naming strategies, wire format, and auto.register.schemas. Framework/config names are version-dependent; treat them as the canonical shape, and verify exact property names against the version you run.
• Apache Avro specification — writer/reader schema resolution, defaults, aliases, type promotion.
• Protocol Buffers language guide — field-number identity, reserved, proto3 defaults, unknown-field preservation.
• Greg Young, Versioning in an Event Sourced System (Leanpub, 2017) — the canonical treatment of upcasting, weak schema, the "never change the meaning of an event" rule, and the Copy and Replace whole-system migration.
• Axon Framework Reference — SingleEventUpcaster / EventUpcasterChain as the JVM reference implementation of ES upcasting.
• Danilo Sato, "Parallel Change" (on martinfowler.com; a.k.a. expand-and-contract) — the universal safe-change recipe in §8.
• Companion guides in this folder: eda-vs-event-sourcing-study-guide.md, event-sourcing-study-guide.md §9, kafka-eda-study-guide.md, inbox-outbox-pattern.md, API-Design-Best-Practices.md.