29 min read
Event-Driven Systems6,379 words

Change Data Capture (CDC)

CDC is the practice of capturing every row-level change committed to a database and emitting it as an event, in commit order, without the application having to publish anything. The modern form reads the database's own transaction log (Postgres WAL, MySQL binlog) with a tool like Debezium and streams the changes to Kafka. It's how you replicate data, build read models, invalidate caches, feed search indexes and warehouses, and relay the transactional outbox — all off a single source of truth: the log the database already writes to survive a crash.

Companion to kafka-eda-study-guide.md, inbox-outbox-pattern.md, and CQRS-comprehensive-guide.md. The outbox guide introduces CDC at §2.4 as the "modern relay" and mentions Debezium's Outbox Event Router; this is the standalone deep dive on the mechanism itself — WAL and logical decoding, Debezium + Kafka Connect internals, snapshots, the change-event envelope, consuming CDC in Spring, the full set of use cases, and the Postgres operational traps that actually page you at 3am. Where CDC powers the outbox specifically, this guide cross-references rather than repeats.

Code convention: application-side snippets (consumers, embedded engine) are shown in Kotlin and Java (Spring Boot 3.x / Spring Kafka, Jakarta namespace). Infrastructure — postgresql.conf, SQL, Kafka Connect JSON, the event envelope — is language-neutral and shown once.

Version note (verified July 2026): Debezium 3.6 is the current stable release (3.6 released 2026-07-01; 3.5 also stable; 2.7 is the last of the 2.x line). Debezium 3.x connectors need Java 17+ (21+ for Debezium Server / Outbox / Quarkus extension) and Kafka Connect 3.1+. PostgreSQL 17 added built-in logical-slot failover. Config-name and snapshot-mode facts below are flagged where they changed across versions.

Organized as: the mental model → the families of CDC → how log-based CDC works on Postgres → Debezium & Kafka Connect → the change-event envelope → snapshots → consuming CDC in Spring → what CDC is for → operating it in production → failure modes → interview framing.


1. The mental model: the database is already an event log#

Every durable database already keeps an ordered, append-only log of every change that ever committed — Postgres calls it the Write-Ahead Log (WAL), MySQL the binlog, SQL Server the transaction log. It exists for crash recovery and replication: the DB writes the change to the log before it touches the data files, so it can always replay the log to reconstruct state. That log is, structurally, an event stream of every insert, update, and delete, in exact commit order.

CDC is nothing more than reading that log and republishing it. That single fact is where every property of CDC comes from:

  • The events can't disagree with the database, because they are the committed changes — derived from the log, not a second write the application has to remember to make. This is why CDC sidesteps the dual-write problem (inbox-outbox §1) by construction: there is no second write.
  • Nothing escapes it. A change made by a batch job, a psql session, an admin's manual UPDATE, or a different application all land in the log the same way. Application-level event publishing only ever sees changes that went through that application.
  • It's low-overhead, because the DB was going to write the log anyway; CDC is a reader, not an extra writer on the hot path.
  • It captures deletes and before-images — things a "poll the table" approach fundamentally cannot see after the fact.

The one-sentence version to say out loud in an interview: "CDC treats the transaction log as the source of truth for change events — the database is already an event-sourced system internally, and CDC is the API to its log." Everything else is mechanism.

The corollary that matters for architecture: CDC inverts who is responsible for the event. Without CDC, the application must actively publish ("I saved an order, now I must send an OrderPlaced"). With CDC, the application just commits its transaction and the event is a consequence. That's powerful (reliability for free) and dangerous (your table schema can leak into your event schema) — the central design tension we return to in §8.


2. The families of CDC: query-based vs log-based#

Not all CDC reads the log. There are three mechanisms, and knowing the trade-offs is table stakes.

2.1 Query-based (polling)#

Poll the table on a schedule for rows that changed since last time, using a monotonic column — updated_at > :lastSeen or a version counter.

sql
SELECT * FROM orders WHERE updated_at > :lastSeenTimestamp ORDER BY updated_at;

Simple, needs no special privileges, works on any database. But it's riddled with holes:

  • Misses deletes entirely — a deleted row doesn't show up in a query; it's just gone. You'd need soft-deletes (deleted_at) to capture them, which changes your data model.
  • Misses intermediate states. If a row changes 5 times between polls, you see only the final state. For an audit trail or an event-per-change consumer, four events vanished.
  • Latency vs. load trade-off. Poll often → low latency, high query load. Poll rarely → cheap, laggy. You can't win both.
  • Clock and boundary hazards. Timestamp ties, clock skew, and rows committed just after your lastSeen cutoff but with an earlier timestamp (the same out-of-order-commit gap the outbox guide warns about in §2.3) cause silently skipped rows.
  • Blind to out-of-band changes unless every writer maintains updated_at perfectly (triggers help, but now see below).

2.2 Trigger-based#

Install database triggers that write every change into a shadow/audit table; a reader tails that table.

  • Captures deletes and before-images (the trigger sees OLD and NEW).
  • But it doubles writes on the hot path (every business write fires a trigger write), couples your schema to trigger logic, is a maintenance burden as tables evolve, and adds latency to the very transactions you care about. Most teams have retired trigger-based CDC in favor of log-based.

2.3 Log-based (the modern default)#

Read the transaction log directly. This is what Debezium does.

  • Complete: every insert, update, delete, in commit order, including before-images and deletes.
  • Low overhead: no extra writes, no triggers, no polling load — it reads a log the DB already produces.
  • Low latency: changes appear as fast as the log is flushed, typically sub-second.
  • Captures out-of-band changes: anything that commits is captured, regardless of which client made it.
  • Cost: it's database-specific (WAL vs binlog vs Oracle redo), needs replication privileges and configuration, and carries operational weight (a connector to run, replication slots to monitor). This is the trade you're buying.
DimensionQuery-based (poll)Trigger-basedLog-based (Debezium)
Captures deletesNo (needs soft-delete)YesYes
Captures every intermediate stateNoYesYes
Before-image availableNoYesYes (with REPLICA IDENTITY)
Load on sourceQuery load, scales badlyWrite amplificationMinimal (log reader)
LatencyPoll intervalLowLow (sub-second)
Out-of-band changesMissedCapturedCaptured
Setup costTrivialModerate, invasiveDB config + connector ops
DB-agnosticYesMostlyNo (per-DB)

The senior framing: query-based CDC is a workaround for not having log access; log-based CDC is the real thing. Reach for polling only when you genuinely can't get logical replication (locked-down managed DB, no privileges) — and even then, prefer the transactional outbox with a polling relay (inbox-outbox §2.3) over polling arbitrary business tables, because at least then you're polling deliberate events, not reverse-engineering intent from row diffs.


3. How log-based CDC works on PostgreSQL#

This is the section that separates people who've run CDC from people who've read a blog post. Postgres CDC rests on four concepts: the WAL, logical decoding, replication slots, and publications — plus one setting that bites everyone, REPLICA IDENTITY.

3.1 WAL and wal_level = logical#

The WAL is Postgres's physical redo log. By default (wal_level = replica) it records enough to recover and to feed physical streaming replicas (byte-for-byte block changes) — but not enough to reconstruct logical row changes like "column status on order 42 went from NEW to PAID." To get that, you must raise the level:

conf
# postgresql.conf — requires a restart wal_level = logical max_wal_senders = 10 # walsender processes (one per replication connection) max_replication_slots = 10 # must be ≥ number of CDC slots you'll create

wal_level = logical makes the WAL carry the extra metadata logical decoding needs. It slightly increases WAL volume; that's the price of CDC.

3.2 Logical decoding and output plugins#

Logical decoding is the Postgres feature that turns the physical WAL into a stream of logical change events, per transaction, in commit order. It hands those changes to an output plugin that formats them:

  • pgoutput — the standard plugin, built into PostgreSQL 10+, maintained by the Postgres project (it's what native logical replication uses). No installation required. This is what you almost always use.
  • decoderbufs — a Protobuf-based plugin maintained by the Debezium community; must be installed on the server.
  • wal2json — an older JSON plugin, largely superseded.

Interview gotcha worth knowing: Debezium's plugin.name config defaults to decoderbufs, but decoderbufs requires installing a plugin on the database. Because pgoutput ships with Postgres and needs nothing extra, virtually every real deployment sets plugin.name=pgoutput explicitly. "The default is decoderbufs, but you use pgoutput" is exactly the kind of detail that signals hands-on experience.

3.3 Replication slots — the double-edged sword#

A replication slot is a named, server-side cursor that remembers how far a specific consumer has confirmed consuming the WAL (confirmed_flush_lsn). Debezium creates one slot and streams through it. The slot gives you durability across connector restarts: stop the connector, restart it a day later, and it resumes exactly where it left off.

That durability guarantee is also the single most dangerous thing about Postgres CDC:

A replication slot pins the WAL. Postgres will not recycle WAL segments the slot hasn't confirmed. If your connector is down (crashed, misconfigured, paused for a deploy) the slot stops advancing and WAL accumulates on the primary's disk until the connector catches up — or the disk fills and the database halts. More than one production Postgres outage traces to a forgotten, inactive Debezium slot. Monitor pg_replication_slots (watch active, and the WAL retained behind restart_lsn) and alert on lag. Postgres 13+ offers max_slot_wal_keep_size as a safety valve — it caps how much WAL a slot may retain and invalidates the slot if exceeded (which then forces a re-snapshot, but saves the database).

An inactive slot is a liability, not a harmless leftover. Dropping obsolete slots is real operational hygiene.

3.4 Publications (what pgoutput streams)#

With pgoutput, the set of tables being captured is defined by a Postgres publication — the same object native logical replication uses:

sql
-- Capture only the tables you mean to; avoid FOR ALL TABLES in production. CREATE PUBLICATION dbz_orders_pub FOR TABLE public.orders, public.order_items;

Debezium can create the publication for you (publication.autocreate.mode = filtered creates one scoped to table.include.list; all_tables mirrors FOR ALL TABLES; disabled means you pre-create it). Pre-creating with a scoped list is the safe default — FOR ALL TABLES captures everything, including tables you never wanted on Kafka, and requires superuser.

3.5 REPLICA IDENTITY — the before-image control (top ops gotcha)#

When an update or delete happens, what does the before image contain? That's governed per-table by REPLICA IDENTITY:

Settingbefore image containsNotes
DEFAULTPrimary-key columns onlyThe default. Updates/deletes carry only the PK as the "old" value. A table with no PK can't emit usable update/delete events.
FULLAll columns' previous valuesWhat you want for full before-images and reliable deletes — but Postgres writes more to the WAL for every update.
USING INDEX idxColumns of a chosen unique indexMiddle ground; the index must be unique and not partial.
NOTHINGNo previous values at allUpdates/deletes carry no old data; usually a mistake for CDC.
sql
ALTER TABLE public.orders REPLICA IDENTITY FULL; -- full before-images for orders

Why it matters in practice: with DEFAULT, a downstream consumer that needs "what was the old status?" simply can't have it — the before-image only has the PK. And it interacts nastily with TOAST (§9.5): large unchanged values show up as a placeholder unless REPLICA IDENTITY FULL. The senior move is to set REPLICA IDENTITY FULL deliberately on tables whose before-image you consume, and to accept the extra WAL cost as a conscious trade.

3.6 The Postgres-side setup, end to end#

sql
-- 1. A least-privilege role that can replicate and read the captured tables. CREATE ROLE debezium WITH REPLICATION LOGIN PASSWORD '***'; GRANT USAGE ON SCHEMA public TO debezium; GRANT SELECT ON public.orders, public.order_items TO debezium; -- for the initial snapshot -- 2. The publication (scoped, not FOR ALL TABLES). CREATE PUBLICATION dbz_orders_pub FOR TABLE public.orders, public.order_items; -- 3. Before-images where you need them. ALTER TABLE public.orders REPLICA IDENTITY FULL;

Plus wal_level = logical in postgresql.conf from §3.1. On managed Postgres (RDS, Cloud SQL, Aurora) the equivalents are a parameter-group setting (rds.logical_replication = 1) and a grant of the provider's replication role — the concepts are identical, the knobs are provider-specific.


4. Debezium and Kafka Connect: the runtime#

Debezium is the de-facto open-source CDC engine. It implements the per-database log readers; the surrounding runtime is usually Kafka Connect.

4.1 Kafka Connect in one paragraph#

Kafka Connect is a distributed framework for moving data in and out of Kafka via connectors. Source connectors pull from external systems into Kafka; sink connectors push Kafka into external systems. Debezium is a source connector. Connect runs a cluster of workers; you POST connector configs to its REST API, and it distributes tasks across workers, restarts failed ones, and stores each connector's progress (for Debezium, the log position) in internal offset / config / status topics. Run Connect in distributed mode (≥2 workers) for HA — if a worker dies, the connector's task is reassigned and resumes from the committed offset.

4.2 A real Postgres connector config#

json
{ "name": "orders-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "debezium", "database.password": "${file:/secrets:pg_password}", "database.dbname": "shop", "topic.prefix": "shop", // v2.x+ name; was "database.server.name" in 1.x "plugin.name": "pgoutput", // NOT the default (decoderbufs) — set it explicitly "slot.name": "debezium_orders", "publication.name": "dbz_orders_pub", "publication.autocreate.mode": "filtered", "table.include.list": "public.orders,public.order_items", "snapshot.mode": "initial", "tombstones.on.delete": "true", "heartbeat.interval.ms": "10000", // see §9.2 — critical on low-traffic DBs "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false" // drop the verbose schema wrapper from the JSON } }
  • topic.prefix names the topics: each table becomes its own topic, topic.prefix.schema.tableshop.public.orders. (In Debezium 1.x this key was database.server.name; flag the rename if you're reading old material.)
  • One connector = one replication slot = one logical stream. You don't parallelize a single connector's streaming; you scale by sharding tables across multiple connectors (each with its own slot and publication). See §9.6.
  • For Avro instead of JSON, point the converters at io.confluent.connect.avro.AvroConverter with a Schema Registry — compact, schema-enforced, the usual production choice.

4.3 Three ways to deploy Debezium#

DeploymentWhat it isUse when
Kafka Connect (standard)Debezium runs as a source connector in a Connect cluster; changes land on Kafka topics.You're on Kafka and want HA, scaling, the connector ecosystem. The default.
Debezium ServerA standalone runtime that streams changes to a non-Kafka sink (Kinesis, Pub/Sub, Pulsar, Redis Streams, HTTP, …).You want CDC without running Kafka/Connect, or your sink isn't Kafka.
Embedded Debezium EngineA Java library you run inside your own app — no Connect, no Kafka; you get a callback per change.You want CDC inside a single Spring service and are willing to own offset storage and HA yourself (§7.3).

5. Anatomy of a change event#

Every Debezium change event is an envelope with a consistent shape. Understanding it is essential to consuming it. Here's a create (insert) event value, JSON, schema wrapper stripped:

json
{ "op": "c", "ts_ms": 1751971200123, "before": null, "after": { "id": 42, "status": "NEW", "total_cents": 5000, "customer_id": 7 }, "source": { "connector": "postgresql", "db": "shop", "schema": "public", "table": "orders", "lsn": 34359739048, "txId": 1201, "snapshot": "false", "ts_ms": 1751971200120 } }

The fields:

  • op — the operation: c create/insert, u update, d delete, r read (a row emitted during the initial snapshot, §6), t truncate.
  • before / after — row state before and after. after is null for deletes; before is null for inserts; for updates you get both (subject to REPLICA IDENTITY, §3.5).
  • source — provenance: connector, db, schema, table, the WAL lsn (log sequence number — a total order token), transaction id, and whether this came from a snapshot. This block is gold for idempotency and ordering (§7.2).
  • ts_ms — when the connector processed the change. (Debezium 3.x also emits higher-resolution ts_us / ts_ns.) Note source.ts_ms (commit time in the DB) differs from top-level ts_ms (capture time) — the gap between them is your CDC lag, and a useful thing to monitor.

An update carries op:"u", both before and after; a delete carries op:"d", populated before, after:null.

Tombstones. After a delete event, Debezium by default (tombstones.on.delete=true) emits a second record with the same key and a null value. This is the signal Kafka log compaction uses to eventually drop the key entirely. If you consume with a plain deserializer, be ready for null-valued records and skip them; if you want compaction to purge deleted keys, keep tombstones on.

Flattening with ExtractNewRecordState. Many consumers don't want the envelope — they want the row. Debezium ships the io.debezium.transforms.ExtractNewRecordState Single Message Transform (SMT) that flattens the event down to just the after state (and turns deletes into tombstones or adds an __deleted flag), so downstream sees a plain row shape:

json
"transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false" // delete handling is controlled by an additional option whose name has shifted across // versions (delete.handling.mode → delete.tombstone.handling.mode in 2.5+); check your version.

Use the flattened form when a consumer just wants current state (a search index, a cache); keep the full envelope when you need before, op, or source (audit, diffs, conditional logic).


6. Snapshots — bootstrapping existing data#

The log only contains recent changes; rows written before CDC started (or before the current WAL retention window) aren't in it. So Debezium begins with a snapshot: it reads the current contents of each captured table and emits every row as a synthetic op:"r" ("read") event, then switches to streaming from the log position captured at snapshot start. The result is a complete history: full current state, then every subsequent change.

6.1 Snapshot modes#

Controlled by snapshot.mode (values as of Debezium 3.x):

ModeBehavior
initial (default)Snapshot on first start (no stored offset), then stream. On restart with an offset, skip the snapshot and just stream.
initial_onlySnapshot, then stop — don't stream. For one-off bulk loads.
no_dataCapture schema only, no rows, then stream. Renamed from schema_only in 2.x — old configs used schema_only.
neverSkip the snapshot; stream from the current log position. You lose pre-existing rows.
alwaysSnapshot on every start. Rare; expensive.
when_neededSnapshot only if there's no offset or the stored offset points at a WAL position the server no longer has. A good safety mode.
recoveryRebuild a lost internal schema-history topic from the DB. Recovery tool, not routine.
configuration_based / customFine-grained control via properties, or your own Snapshotter SPI implementation.

6.2 Consistency without long locks#

A naive snapshot would need to lock the whole table so it doesn't shift under you. Postgres avoids that: the connector opens a REPEATABLE READ transaction with an exported snapshot, so it reads a single consistent point-in-time view of the data while concurrent writes proceed, and it knows the exact LSN at which to begin streaming so nothing is missed or duplicated across the snapshot→stream handoff. (MySQL historically needed a brief global read lock; Postgres's MVCC makes this cheaper.)

6.3 The big-table problem → incremental snapshots#

A classic snapshot has two operational teeth: on a huge table it can run for hours, during which streaming hasn't started (lag builds); and if it fails at 90%, older Debezium restarted it from scratch. Incremental snapshots fix both. Based on Netflix's DBLog watermark algorithm, they:

  • Read the table in chunks, interleaved with live streaming — so change events keep flowing during the snapshot instead of waiting for it.
  • Are resumable — a restart continues from the last completed chunk, not the beginning.
  • Can be triggered ad hoc at runtime by writing to a signaling table (signal.data.collection) with an execute-snapshot signal — so you can (re)snapshot a newly added table, or backfill after changing table.include.list, without restarting the connector or re-snapshotting everything.

"Use incremental snapshots for large tables and for adding tables to a running connector" is a strong, specific senior answer to "how do you onboard a 500M-row table without downtime?"


7. Consuming CDC events in Spring Boot#

You consume Debezium topics like any Kafka topic — but the payload is a change envelope, and the delivery guarantee is at-least-once (Kafka Connect commits source offsets after delivering, so a worker crash can redeliver). That means your consumer must be idempotent — the same discipline as the inbox-outbox guide, applied to CDC.

7.1 A @KafkaListener on the change topic#

Consuming the flattened form (after ExtractNewRecordState) to maintain a projection. We use an idempotent upsert so redelivery converges (inbox-outbox §3.6):

Kotlin

kotlin
data class OrderRow(val id: Long, val status: String, val totalCents: Long, val customerId: Long) @Component class OrderProjectionListener( private val projection: OrderProjectionRepository, ) { @KafkaListener(topics = ["shop.public.orders"], groupId = "order-projection") fun onChange( @Payload(required = false) row: OrderRow?, // null value == tombstone (deleted key) @Header(KafkaHeaders.RECEIVED_KEY) key: String, ) { if (row == null) { // delete/tombstone projection.deleteByOrderId(key.toLong()) return } projection.upsert(row.id, row.status, row.totalCents, row.customerId) // idempotent } }

Java

java
public record OrderRow(long id, String status, long totalCents, long customerId) {} @Component public class OrderProjectionListener { private final OrderProjectionRepository projection; // constructor omitted @KafkaListener(topics = "shop.public.orders", groupId = "order-projection") public void onChange( @Payload(required = false) OrderRow row, // null value == tombstone (deleted key) @Header(KafkaHeaders.RECEIVED_KEY) String key) { if (row == null) { // delete/tombstone projection.deleteByOrderId(Long.parseLong(key)); return; } projection.upsert(row.id(), row.status(), row.totalCents(), row.customerId()); // idempotent } }

Consuming the full envelope instead, to branch on the operation and use the before-image:

Kotlin

kotlin
data class Envelope(val op: String, val before: JsonNode?, val after: JsonNode?, val source: JsonNode) @KafkaListener(topics = ["shop.public.orders"], groupId = "order-audit") fun onEnvelope(@Payload envelope: Envelope) { when (envelope.op) { "c", "r" -> auditService.recordCreated(envelope.after!!) // r = snapshot read "u" -> auditService.recordChange(envelope.before!!, envelope.after!!) "d" -> auditService.recordDeleted(envelope.before!!) "t" -> auditService.recordTruncate(envelope.source) } }

Java

java
public record Envelope(String op, JsonNode before, JsonNode after, JsonNode source) {} @KafkaListener(topics = "shop.public.orders", groupId = "order-audit") public void onEnvelope(@Payload Envelope envelope) { switch (envelope.op()) { case "c", "r" -> auditService.recordCreated(envelope.after()); // r = snapshot read case "u" -> auditService.recordChange(envelope.before(), envelope.after()); case "d" -> auditService.recordDeleted(envelope.before()); case "t" -> auditService.recordTruncate(envelope.source()); } }

7.2 Idempotency and ordering with CDC#

  • Per-key ordering is guaranteed; global ordering is not. Debezium keys each record by the table's primary key, so all changes to one row land in one partition in commit order. Changes across different rows/tables can interleave. Don't build logic that assumes a global order (inbox-outbox §3.7).
  • Use the LSN as a version/idempotency token. source.lsn is monotonic in commit order. Persisting the last-applied LSN per key lets a projection ignore stale replays (WHERE incoming_lsn > stored_lsn) — this makes reprocessing after a snapshot or offset reset safe, and turns "at-least-once" into an idempotent effect without a separate inbox table.
  • Snapshot r events replay your whole table. If a connector re-snapshots (slot lost, mode always, manual reset), every row arrives again as op:"r". Idempotent upserts (or the LSN guard) absorb this; non-idempotent effects (send an email per event) would fire for the entire table. Design consumers so a re-snapshot is a no-op, not a catastrophe.

7.3 Embedded Debezium — CDC without Kafka Connect#

When you want CDC inside a single Spring service and don't want to run Connect, use the Embedded Debezium Engine. You get a callback per change; you own offset storage and the fact that this runs on exactly one instance (no built-in HA — use leader election if you run replicas).

Kotlin

kotlin
@Component class EmbeddedCdc(private val handler: ChangeHandler) { private val executor = Executors.newSingleThreadExecutor() private lateinit var engine: DebeziumEngine<ChangeEvent<String, String>> @PostConstruct fun start() { val props = Properties().apply { put("name", "embedded-orders") put("connector.class", "io.debezium.connector.postgresql.PostgresConnector") put("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore") put("offset.storage.file.filename", "/data/offsets.dat") put("offset.flush.interval.ms", "5000") put("database.hostname", "postgres"); put("database.port", "5432") put("database.user", "debezium"); put("database.password", "***") put("database.dbname", "shop"); put("topic.prefix", "shop") put("plugin.name", "pgoutput"); put("slot.name", "embedded_orders") put("table.include.list", "public.orders") } engine = DebeziumEngine.create(Json::class.java) .using(props) .notifying { record -> handler.handle(record.key(), record.value()) } // value() is JSON .build() executor.execute(engine) } @PreDestroy fun stop() { engine.close(); executor.shutdown() } // close() flushes offsets — don't skip it }

Java

java
@Component public class EmbeddedCdc { private final ChangeHandler handler; private final ExecutorService executor = Executors.newSingleThreadExecutor(); private DebeziumEngine<ChangeEvent<String, String>> engine; // constructor omitted @PostConstruct public void start() { Properties props = new Properties(); props.setProperty("name", "embedded-orders"); props.setProperty("connector.class", "io.debezium.connector.postgresql.PostgresConnector"); props.setProperty("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore"); props.setProperty("offset.storage.file.filename", "/data/offsets.dat"); props.setProperty("offset.flush.interval.ms", "5000"); props.setProperty("database.hostname", "postgres"); props.setProperty("database.port", "5432"); props.setProperty("database.user", "debezium"); props.setProperty("database.password", "***"); props.setProperty("database.dbname", "shop"); props.setProperty("topic.prefix", "shop"); props.setProperty("plugin.name", "pgoutput"); props.setProperty("slot.name", "embedded_orders"); props.setProperty("table.include.list", "public.orders"); engine = DebeziumEngine.create(Json.class) .using(props) .notifying(record -> handler.handle(record.key(), record.value())) // value() is JSON .build(); executor.execute(engine); } @PreDestroy public void stop() throws IOException { engine.close(); executor.shutdown(); } // close() flushes offsets }

The trade: no Kafka/Connect to operate, but you inherit offset durability, restart safety, and single-writer coordination. Good for small footprints and for streaming into a sink that isn't Kafka; not a substitute for Connect at scale.


8. What CDC is for — the use cases#

CDC's reputation as "just the outbox relay" undersells it. The same log stream powers a whole family of patterns. The unifying idea: anywhere you need a second system to stay consistent with a database, CDC keeps it in sync without the writing service knowing that system exists.

Use caseWhat CDC doesWhy it fits
Replication / data syncStream one DB's changes into another (same or different engine).The original CDC use; keeps replicas/forks current in near-real-time.
Read models / CQRSMaterialize query-optimized views from the write model's changes.Decouples read and write models; the projection updates itself (CQRS-comprehensive-guide.md).
Cache invalidationEvict/refresh cache entries when the row of record changes.Solves "who invalidates the cache?" — the DB change is the trigger, so caches can't silently go stale.
Search indexingFeed Elasticsearch/OpenSearch from row changes.Keeps the index eventually consistent with the DB without dual-writes from the app.
Analytics / warehouse / lakeStream changes into Snowflake, BigQuery, Iceberg, etc.Near-real-time ELT with minimal load on the OLTP source — no nightly full-table dumps.
Microservices decoupling / stranglerShare data between services, or migrate off a monolith DB, by streaming its changes.Lets a new service consume the old DB's changes without a shared database or a big-bang cutover.
Transactional outbox relayTail an outbox table and publish deliberate domain events.Reliable event publishing with no dual-write (inbox-outbox §2.4).
Audit / historyPersist every before/after as an immutable trail.The log already has every change; CDC just durably records it.

8.1 The central design decision: CDC on business tables vs. on an outbox#

This is the judgment call interviewers probe, because getting it wrong couples your whole architecture:

  • CDC directly on business tables (orders, order_items, …) is the fastest way to a stream, and it's the right tool for replication, analytics, and search indexing — cases where you genuinely want table-shaped data in another store.
  • But it makes your table schema your public event contract. Every consumer now depends on your column names and types. Rename a column and you break downstream systems. You also emit row-shaped changes, not domain events — three related UPDATEs to three tables surface as three unrelated events, with no "OrderCancelled" concept tying them together. And you can't emit an event that isn't a single row change.
  • CDC on a transactional outbox table keeps the reliability (still no dual-write) but restores deliberate, versioned domain events: the application writes a purpose-built OrderPlaced payload into the outbox in the same transaction as the business change, and CDC relays that. Debezium's Outbox Event Router SMT (io.debezium.transforms.outbox.EventRouter) routes by an aggregatetype column and keys by aggregateid, so the relay is connector config, not code (inbox-outbox §2.4).

The senior formulation: "Use raw-table CDC when the consumer legitimately wants the data's table shape — replication, search, analytics. Put an outbox in front of CDC when you're publishing domain events other services react to, because you don't want your internal schema to be their contract." Naming that distinction unprompted is a strong signal.


9. Operating CDC in production — the hard parts#

Everything so far works on a laptop. These are the things that bite at scale.

9.1 Replication-slot WAL growth (the disk-fills outage)#

Covered in §3.3 and worth repeating because it's the #1 Postgres CDC incident: an inactive or lagging slot pins WAL and can fill the primary's disk, halting the database. Monitor pg_replication_slots, alert on a slot going inactive or its retained-WAL growing, cap with max_slot_wal_keep_size (PG13+), and treat abandoned slots as urgent cleanup — dropping a slot from a decommissioned connector is not optional housekeeping.

9.2 The idle-database / low-traffic slot lag (heartbeats)#

A subtle one. A slot's confirmed position only advances when the connector confirms WAL it has processed. If your captured tables are quiet but other tables in the same database are busy, the connector sees no events to confirm, so its confirmed_flush_lsn doesn't advance — even though WAL is piling up from the busy tables. The slot lags, WAL grows, and it looks like §9.1 with no obvious cause.

The fix is heartbeats: set heartbeat.interval.ms, and on a low-traffic capture add a heartbeat.action.query that periodically writes to a dedicated heartbeat table the connector does capture — that generates a captured change, which lets the connector confirm progress and advance the slot. This is a famous Postgres+Debezium gotcha; knowing it cold marks real experience.

9.3 Primary failover and slot survival#

Historically, logical replication slots were not copied to physical standbys. So if your primary failed over to a replica, the slot vanished, and the connector had to re-snapshot from scratch on the new primary — a serious availability event for a big database. PostgreSQL 17 added built-in logical-slot failover/synchronization (sync_replication_slots, a failover flag on the slot, standby_slot_names) so the slot survives promotion; before PG17 you needed an extension like pg_failover_slots or you accepted the re-snapshot. On managed Postgres, check whether the provider syncs slots across failover — several didn't until recently.

9.4 Schema evolution / DDL#

Row changes are easy; schema changes need thought. Adding a nullable column generally flows through cleanly (new events carry the new field). Renames, drops, and type changes can break downstream consumers whose code expects the old shape — this is precisely why an outbox with versioned event payloads (§8.1) is safer for domain events than raw-table CDC. Coordinate producer/consumer schema changes the same way you would any contract change; with Avro + Schema Registry you get compatibility enforcement for free.

9.5 TOAST columns (the placeholder surprise)#

Postgres stores large field values (big text/JSON/bytea) out of line via TOAST. If an UPDATE doesn't change a TOASTed column, Postgres doesn't put its value in the WAL — so Debezium can't see it and emits the placeholder __debezium_unavailable_value instead of the real content. A consumer that blindly writes after will overwrite good data with that sentinel. Fixes: set REPLICA IDENTITY FULL (Postgres then logs the full row, including unchanged TOAST values, at extra WAL cost), or use the reselect-columns post-processor to re-fetch the missing columns from the DB, or teach the consumer to treat the placeholder as "unchanged, keep existing." Surprising the first time; obvious once you know it.

9.6 Throughput and the single-slot ceiling#

One connector streams through one slot, single-threaded. You don't scale it by adding threads; you scale by sharding tables across multiple connectors, each with its own slot and publication. Very high-write tables can also outrun a JSON pipeline — Avro and tuned batch/flush settings help. Know that the scaling unit is "another connector," not "more parallelism in this one."

9.7 Ordering and transaction boundaries#

Per-key order is preserved; cross-table transactional atomicity is not visible by default — two tables changed in one DB transaction land on two topics with no marker tying them together. If a consumer needs to know "these changes were one transaction," enable transaction metadata (provide.transaction.metadata=true), which emits BEGIN/END markers on a separate topic with event counts and ordering. Most consumers don't need it; the ones reconstructing multi-table aggregates do.


10. Failure modes & fixes (Problem → Cause → Fix)#

10.1 Postgres disk fills, database stallsCause: an inactive/lagging replication slot pins WAL (connector down, or §10.2). Fix: monitor pg_replication_slots and alert on inactivity/lag; drop abandoned slots; set max_slot_wal_keep_size (PG13+) as a backstop; fix the connector fast (§9.1).

10.2 Slot lags even though the connector is healthyCause: captured tables are idle while other tables generate WAL, so the connector never confirms progress and confirmed_flush_lsn stalls. Fix: enable heartbeat.interval.ms + heartbeat.action.query writing to a captured heartbeat table (§9.2).

10.3 Full re-snapshot after a database failoverCause: pre-PG17 logical slots weren't synced to standbys, so promotion lost the slot. Fix: PostgreSQL 17 slot failover (sync_replication_slots), or pg_failover_slots on older versions; confirm your managed provider syncs slots (§9.3).

10.4 Consumer overwrites good data with __debezium_unavailable_valueCause: an unchanged TOAST column isn't in the WAL, so Debezium sends a placeholder. Fix: REPLICA IDENTITY FULL, the reselect-columns post-processor, or treat the placeholder as "unchanged" in the consumer (§9.5).

10.5 Update/delete events have no old values (or no PK to key on)Cause: REPLICA IDENTITY DEFAULT carries only the PK; NOTHING carries nothing; a table with no PK can't produce usable update/delete events. Fix: add a primary key; set REPLICA IDENTITY FULL where you consume before-images (§3.5).

10.6 Duplicate CDC events after a restartCause: Kafka Connect is at-least-once — a worker crash after delivery but before the offset commit redelivers. Fix: idempotent consumers — upserts and/or an LSN high-water guard (WHERE incoming_lsn > stored_lsn); the inbox pattern for non-idempotent effects (§7.2, inbox-outbox §3).

10.7 Whole table replays as eventsCause: a (re)snapshot emits every row as op:"r" (mode always, a lost slot, a manual reset). Fix: make consumers idempotent so a re-snapshot is a no-op; never wire non-idempotent side effects directly to a raw CDC topic (§7.2).

10.8 Downstream breaks when a column is renamedCause: raw-table CDC made your table schema the event contract. Fix: publish domain events via an outbox + Outbox Event Router instead of capturing business tables directly; version the payloads (§8.1).

10.9 Events exist but arrive out of order across entitiesCause: expecting global ordering; CDC only guarantees per-key order within a partition. Fix: key by the entity (Debezium does, by PK); use LSN/version checks to drop stale updates; enable transaction metadata if you must reconstruct transaction boundaries (§9.7).

10.10 CDC pipeline can't keep up with a hot tableCause: a single connector/slot is single-threaded. Fix: shard tables across multiple connectors (each its own slot/publication); switch to Avro; tune batch/flush; for a giant table onboard via incremental snapshot so streaming isn't blocked (§6.3, §9.6).

10.11 A newly added table has no historical data on its topicCause: it was added to table.include.list after the initial snapshot, and streaming only carries changes from here on. Fix: trigger an ad-hoc incremental snapshot via the signaling table to backfill it without restarting or re-snapshotting everything (§6.3).


11. Interview framing — how to sound senior#

  • Lead with the mental model. "The database already keeps an ordered, durable log of every change for its own recovery; CDC just reads that log and republishes it. So the events are derived from committed state — they can't disagree with the database, which is exactly why CDC avoids the dual-write problem." That one framing shows you understand why CDC works, not just that Debezium exists.
  • Know the query-based vs log-based trade-off and say log-based is the real thing: it captures deletes, intermediate states, and out-of-band changes at low overhead, at the cost of DB-specific setup and ops. Reach for polling only when you can't get log access.
  • Show the Postgres mechanism: wal_level=logical → logical decoding → an output plugin (pgoutput, built into PG10+ — even though Debezium's default is decoderbufs) → a replication slot (durable cursor that also pins WAL) → a publication (which tables). Then land the ops punchline: an inactive slot fills the disk. That progression is the whole system in six phrases.
  • Get REPLICA IDENTITY rightDEFAULT gives only the PK in before-images; FULL gives all columns at extra WAL cost; it interacts with TOAST. This is the detail that separates readers from operators.
  • Nail the delivery guarantee: Debezium/Kafka Connect is at-least-once, so consumers must be idempotent (LSN guard or upsert), same as any Kafka consumer. Don't claim exactly-once.
  • Make the outbox-vs-raw-table call explicitly: raw-table CDC for replication/search/analytics where you want table-shaped data; outbox + Event Router when you're publishing domain events other services depend on, so your internal schema isn't their contract. This is the highest-signal architectural point in the whole topic.
  • Have three war stories ready: the disk-filling inactive slot; the idle-DB slot lag fixed with heartbeats; the failover that forces a re-snapshot (and PG17 fixing it). Specific failure modes with specific fixes read as scar tissue, not textbook.
  • Mention snapshots and incremental snapshots: the initial snapshot emits rows as op:"r"; incremental (DBLog watermark) snapshots are chunked, resumable, and triggerable at runtime to onboard huge or newly added tables without downtime.
  • Attribution that lands: Debezium is the reference open-source CDC engine (Red Hat), runs on Kafka Connect; Martin Kleppmann's "turning the database inside out" and the log-as-source-of-truth framing (also Jay Kreps's "The Log") are the canonical intellectual lineage; the incremental-snapshot algorithm is Netflix's DBLog.

12. Cheat-sheet#

ThingWhat to remember
CDC in one lineRead the DB's transaction log and republish every row change as an event, in commit order.
Why it beats app-level publishingEvents are derived from committed state → no dual-write; captures deletes, intermediate states, and out-of-band changes.
FamiliesQuery-based (polling — misses deletes/intermediate states), trigger-based (write amplification), log-based (Debezium — the real thing).
Postgres prerequisiteswal_level = logical, max_wal_senders/max_replication_slots > 0, a REPLICATION-privileged role.
Logical decoding pluginUse pgoutput (built into PG10+, no install). Debezium's plugin.name default is decoderbufs — set pgoutput explicitly.
Replication slotDurable per-consumer WAL cursor. Pins WAL → an inactive slot fills the disk. Monitor pg_replication_slots; max_slot_wal_keep_size (PG13+) as backstop.
PublicationWhich tables pgoutput streams. Prefer a scoped CREATE PUBLICATION, not FOR ALL TABLES.
REPLICA IDENTITYDEFAULT = PK only in before-image; FULL = all columns (extra WAL); NOTHING = nothing; needed for real before-images + deletes.
Event envelopeop (c/u/d/r snapshot/t truncate), before, after, source (incl. lsn), ts_ms.
Delivery guaranteeAt-least-once (Kafka Connect commits offsets after delivery). Consumers must be idempotent — upsert or LSN high-water guard.
Snapshot modesinitial (default), initial_only, no_data (was schema_only), never, always, when_needed, recovery.
Incremental snapshotChunked, resumable, concurrent with streaming; triggered via a signaling table. Onboards huge/new tables with no downtime (Netflix DBLog).
Topic namingtopic.prefix.schema.table, one topic per table. topic.prefix was database.server.name pre-2.0.
Flatten SMTExtractNewRecordState → just the after row. Outbox Event Router → domain events from an outbox table.
Raw-table vs outbox CDCRaw tables for replication/search/analytics (want table shape); outbox for domain events (don't leak your schema as the contract).
Deploy optionsKafka Connect (default), Debezium Server (non-Kafka sinks), Embedded Engine (in-process, you own offsets/HA).
Top prod trapsInactive slot fills disk (§9.1); idle-DB slot lag → heartbeats (§9.2); failover loses slot pre-PG17 (§9.3); unchanged TOAST → __debezium_unavailable_value (§9.5).
Version facts (Jul 2026)Debezium 3.6 stable; Java 17+ (21+ for Server/Outbox); Kafka Connect 3.1+; PG17 adds logical-slot failover.

Quick self-test: Why does log-based CDC avoid the dual-write problem entirely, and what can it capture that query-based polling can't? What are the four Postgres concepts CDC rests on, and which one can fill your disk? Why is pgoutput what you use even though Debezium's plugin.name default is decoderbufs? What does REPLICA IDENTITY FULL change, and why would you pay its cost? What delivery guarantee does Debezium give, and how do you make a consumer idempotent under it? When do you capture business tables directly, and when do you put an outbox in front of CDC? Your captured tables are idle but the slot keeps lagging and WAL keeps growing — what's happening and what's the fix? A downstream team says a newly added table has no history on its topic — what do you do without restarting the connector? If you can answer all eight, you're ready.


Sources / lineage: Debezium documentation (PostgreSQL connector, snapshot modes, Outbox Event Router, ExtractNewRecordState, reselect-columns) as of Debezium 3.6 (stable, July 2026); PostgreSQL 17 logical-replication-failover documentation; Netflix DBLog paper (incremental snapshots); Kleppmann, "Turning the Database Inside Out" and Kreps, "The Log." Verify version-specific config names (topic.prefix vs database.server.name; no_data vs schema_only) against the exact Debezium version you deploy.