Kafka & Event-Driven Architecture

A reference for system-design and deep-dive interviews. Organized as: mental model → best practices → EDA patterns → failure modes and fixes → interview framing → config cheat-sheet.

Defaults noted here are for modern Kafka (3.0+), where several important defaults changed. Kafka 4.0 (2025) removed ZooKeeper entirely; KRaft is now the metadata quorum. Flag version-dependence when you cite a default in an interview.

1. The mental model (get this right and everything follows)#

Kafka is a distributed, replicated, append-only commit log, not a message queue. The distinction drives almost every design decision.

• Topic: a named stream of records. Split into partitions.
• Partition: the unit of parallelism, ordering, and replication. An ordered, immutable sequence of records, each with a monotonically increasing offset. Ordering is guaranteed only within a partition, never across a topic.
• Broker: a server holding some partitions. A cluster is many brokers.
• Leader / follower replicas: each partition has one leader (handles all reads/writes) and N-1 followers replicating it. Replication factor (RF) is usually 3.
• ISR (in-sync replicas): the subset of replicas caught up within replica.lag.time.max.ms. Durability guarantees are defined against the ISR, not all replicas.
• Producer: appends records to partitions, choosing the partition by key hash (or round-robin/sticky if keyless).
• Consumer group: a set of consumers that share the work of a topic. Each partition is consumed by exactly one consumer in the group at a time — so max useful consumer parallelism = partition count.
• Offset: the consumer's bookmark. Kafka stores committed offsets in the internal __consumer_offsets topic. Consumers control their own position — this is why replay, reprocessing, and multiple independent consumers of the same topic are trivial. This is the key difference from a traditional broker that deletes on ack.
• Retention: records persist by time/size (retention.ms, retention.bytes), independent of consumption. A consumer can be down for hours and catch up. Compacted topics retain the latest value per key instead.

The one-line framing for interviews: "Kafka is a durable, partitioned log where consumers track their own offsets, which decouples producers from consumers in time and lets many independent consumers replay the same stream."

Why the log design matters: it gives you replayability (reset offsets), multiple independent subscribers (each group has its own offsets), high throughput (sequential disk I/O + zero-copy + batching), and durability (replication) — all from the same primitive.

2. Delivery & processing semantics#

This is the single most-tested area. Be precise.

At-most-once: commit offset before processing. Crash → message lost. Rarely what you want.

At-least-once (the default posture): process, then commit. Crash after processing but before commit → reprocessing → duplicates. You handle duplicates with idempotent consumers (see below). This is the pragmatic default for most systems.

Exactly-once (EOS): Kafka provides this within Kafka via two mechanisms:

• Idempotent producer (enable.idempotence=true, default in 3.0+): the broker deduplicates retries using a producer ID + per-partition sequence number, so producer retries don't create duplicates. Eliminates duplicates from the producer path and prevents reordering on retry.
• Transactions (transactional.id): atomically write to multiple partitions/topics and commit consumer offsets in one transaction. Consumers set isolation.level=read_committed to only see committed records. This gives exactly-once for read-process-write loops inside Kafka (the Kafka Streams processing.guarantee=exactly_once_v2 model).

The critical caveat interviewers probe: EOS does not magically extend to external systems. If your consumer writes to Postgres or calls a payment API, Kafka transactions don't cover that. For end-to-end correctness you need either (a) idempotent writes to the external system, (b) the transactional outbox pattern, or (c) deduplication keyed on a business/message ID. Saying "we'll just turn on exactly-once" for an external sink is the classic wrong answer.

Practical stance to state in an interview: "I default to at-least-once with idempotent consumers, and reserve Kafka transactions/EOS for pure Kafka-to-Kafka stream processing. For external side effects I make the operation idempotent rather than relying on EOS."

3. Ordering & partitioning#

• Ordering guarantee: per-partition only. If you need ordering for an entity (e.g., all events for account-123), use a partition key so all its records hash to the same partition.
• Consequence: global ordering across a topic requires a single partition — which caps throughput at one consumer. Almost always the wrong trade; design so you only need per-key ordering.
• Idempotent producer preserves order on retries (with max.in.flight.requests.per.connection ≤ 5). Without idempotence, retries can reorder in-flight batches.
• Partition count: the most consequential early decision.
  • You can increase partitions but never decrease. Increasing re-hashes keys → a given key may move to a new partition → breaks per-key ordering across the change and can split state. Treat repartitioning as a migration, not a config tweak.
  • Over-partitioning has real costs: more open file handles, more memory, longer leader-election/failover, longer rebalances, more end-to-end latency.
  • Rule of thumb: size partitions for peak target throughput and future consumer parallelism, e.g. max(target_throughput/per_partition_throughput, desired_consumer_count), then add headroom. Prefer a sensible number over "a huge number just in case."
• Keyless records spread via the sticky partitioner (batches to one partition until full, then rotates) for better batching — good throughput, no ordering guarantee.

4. Producer best practices#

• Durability: acks=all (default in 3.0+) plus min.insync.replicas=2 on a RF=3 topic. acks=all alone is not enough — without min.insync.replicas≥2, a partition with only the leader in ISR still "succeeds," so you can lose data. The pair is what guarantees a write survives one broker loss.
• Idempotence on (enable.idempotence=true, default 3.0+): free dedup of retries + order preservation. Requires acks=all, retries>0, max.in.flight ≤ 5.
• Retries: leave retries high (effectively infinite via delivery.timeout.ms, default 2 min). Bound end-to-end with delivery.timeout.ms rather than a small retry count.
• Batching & compression: tune batch.size and linger.ms (a few ms of linger dramatically improves throughput by filling batches); use compression.type=lz4/zstd. Compression is per-batch, so bigger batches compress better.
• Backpressure: the producer buffers in buffer.memory; when full, send() blocks up to max.block.ms. Monitor buffer exhaustion — it's an early warning of a slow/overloaded cluster.
• Partitioning by key for ordering; be aware of key skew (below).
• Don't fire-and-forget unless you truly accept loss — handle the send callback / await the future so you notice failures.

5. Consumer best practices#

• Offset commits: prefer manual commits after successful processing (enable.auto.commit=false) for at-least-once. Auto-commit (default, every 5s) commits on a timer regardless of whether processing finished → silent message loss on crash.
• The poll loop is a liveness contract: the consumer must call poll() at least every max.poll.interval.ms (default 5 min), or the group coordinator assumes it's dead and rebalances its partitions away. Long per-record processing is the #1 cause of unexpected rebalances (see §7).
  • Fixes: reduce max.poll.records (default 500) so each batch is processable in time; move slow work off-thread with careful offset tracking; or raise max.poll.interval.ms.
• Heartbeats vs poll: heartbeats run on a background thread (heartbeat.interval.ms ~3s, session.timeout.ms default 45s in 3.0+). Session timeout detects a dead consumer; poll interval detects a stuck (livelocked) one. Know the difference.
• Rebalance strategy: use CooperativeStickyAssignor (incremental cooperative rebalancing, KIP-429) instead of eager. Eager rebalancing is "stop-the-world" — every consumer drops all partitions and pauses. Cooperative only revokes the partitions that actually move.
• Static membership (group.instance.id, KIP-345): lets a consumer restart (deploys, rolling upgrades) within session.timeout.ms without triggering a rebalance. Big stability win for k8s rollouts.
• Idempotent processing: since you're at-least-once, make handlers idempotent — dedup on a message/business key (DB unique constraint, upsert, or a seen-set with TTL), or make the downstream write naturally idempotent (SET not INCREMENT).
• Commit granularity: commit periodically/by batch, not per message (per-message commits kill throughput). Balance against how much reprocessing a crash costs you.

6. Replication, durability & availability#

• RF=3 is the standard. Survives one broker loss with no data loss and stays available; survives two with data intact (read-only for that partition until a replica returns).
• min.insync.replicas: the durability knob. With RF=3, set it to 2. This means: tolerate one broker down and still accept writes; if two are down, reject writes (fail fast) rather than risk data on a lone replica. It's a deliberate consistency-over-availability choice (CP-leaning).
• unclean.leader.election.enable=false (default): never elect an out-of-sync replica as leader. true trades durability for availability — an out-of-sync replica becoming leader silently drops the records it never received. Only enable if you truly prefer availability to data (rare).
• Rack awareness (broker.rack + rack-aware assignment): spread replicas across racks/AZs so one zone failure doesn't take all replicas of a partition.
• The durability triangle to recite: acks=all + min.insync.replicas=2 + unclean.leader.election=false + RF≥3. Miss any leg and you have a data-loss window.

7. Schema management & contracts#

Events are a public API with a longer lifetime than any single service. Treat schemas as contracts.

• Use a Schema Registry (Confluent/Apicurio) with Avro, Protobuf, or JSON Schema. Producers register schemas; messages carry a schema ID, not the full schema.
• Compatibility modes: BACKWARD (new consumers read old data — the common default; allows adding optional fields / removing fields with defaults), FORWARD (old consumers read new data), FULL (both). Choose based on whether you upgrade producers or consumers first.
• Evolution rules: add fields with defaults; never rename/retype/remove-required in place; treat a breaking change as a new topic/version, not an edit.
• Why it matters: without enforced schemas, a producer change silently breaks every downstream consumer — the failure shows up far from its cause. Schema Registry moves that failure to publish time where the right team sees it.

8. Event-driven architecture patterns#

8.1 What's actually in the event? (Fowler's taxonomy)#

The most important EDA design choice, and a frequent interview question.

• Event Notification: "something happened," minimal payload (just IDs). Consumers call back to the source for details. Pros: small events, low coupling to source's data model. Cons: chatty (N callbacks), source must stay available, no easy replay of state.
• Event-Carried State Transfer (ECST): the event carries all the data a consumer needs (the full changed entity). Consumers keep their own local read models and never call back. Pros: consumers are autonomous and resilient to source downtime, great for decoupling. Cons: larger events, data duplication, eventual consistency, versioning burden. This is usually the right default for decoupled microservices.
• Event Sourcing: the event log is the source of truth. State is derived by replaying events; you never store just current state. Pros: full audit history, time-travel, rebuild any projection. Cons: real complexity — schema evolution of historical events, snapshotting for performance, "how do I query current state?" (you can't easily), GDPR/right-to-be-forgotten on an immutable log. Don't claim event sourcing when you just mean "we use Kafka." Interviewers love catching this conflation.
• CQRS (Command Query Responsibility Segregation): separate the write model from one or more read models, often fed by events. Pairs naturally with event sourcing but is independent of it. Justified when read and write shapes/scales diverge; overkill for simple CRUD.

8.2 Choreography vs orchestration#

• Choreography: services react to events with no central coordinator. Emergent workflow. Pros: loose coupling, easy to add consumers. Cons: no single place shows the end-to-end flow; hard to reason about, debug, and change; risk of cyclic event chains. Great for small numbers of steps.
• Orchestration: a central orchestrator (or workflow engine — Temporal, Camunda, a saga coordinator) drives the steps. Pros: explicit, observable, easier error handling and timeouts. Cons: the orchestrator is a coupling point and can become a bottleneck/god-service.
• Rule of thumb: choreography for simple, stable flows; orchestration once a business process has many steps, compensations, and timeouts you need to see and control.

8.3 The dual-write problem → Transactional Outbox#

The single most important EDA correctness pattern. If a service does db.save() and kafka.send() as two separate operations, they are not atomic: a crash between them leaves the DB updated but no event published (or vice versa). Retrying naively causes duplicates or inconsistency. There is no correct way to write to two systems atomically with two independent calls.

Solution — Transactional Outbox: in the same local DB transaction as your business write, insert a row into an outbox table. A separate relay publishes those rows to Kafka and marks them sent. The DB transaction makes the business change and the "intent to publish" atomic. Two ways to run the relay:

• Polling publisher: a job reads unsent outbox rows and publishes them. Simple, works anywhere.
• Change Data Capture (CDC): tail the DB transaction log with Debezium → Kafka Connect. No polling, low latency, no app changes to publish. The standard modern approach.

Publishing is at-least-once (relay can crash after publish, before marking sent), so consumers must still dedup on the outbox/event ID. State this explicitly.

8.4 Saga pattern (distributed transactions without 2PC)#

For a business transaction spanning services (order → payment → inventory → shipping), you can't hold a distributed ACID transaction. A saga is a sequence of local transactions, each emitting an event that triggers the next; if a step fails, you run compensating transactions to semantically undo prior steps (refund, restock).

• Choreographed saga: each service listens and reacts. Simple, but the flow is implicit.
• Orchestrated saga: a saga orchestrator issues commands and tracks state. Preferred for complex flows.
• Key properties to mention: no isolation (intermediate states are visible → design for it), compensations must be idempotent and are semantic not literal rollbacks, and you need timeouts + a "stuck saga" strategy.

8.5 Topic design & data lifecycle#

• Naming: adopt a convention (<domain>.<entity>.<event-type>, e.g. orders.order.created). Version in the name or schema, not ad hoc.
• Retention topics (cleanup.policy=delete): time/size-bounded event streams — the default for facts/notifications.
• Compacted topics (cleanup.policy=compact): retain the latest value per key forever → perfect for "current state" streams, changelogs, and building local caches/materialized views. A tombstone (null value) deletes a key. Compaction is eventual (runs on closed segments), so old values linger briefly — don't rely on immediate removal.
• compact,delete combined: keep latest-per-key and age out old keys.
• Number of topics vs partitions: prefer fewer topics with more partitions over thousands of tiny topics; per-partition overhead is real.

9. Problems & how to solve them#

The heart of the "what goes wrong" question. Each is Problem → Root cause → Fix.

9.1 Consumer lag (falling behind)#

Symptom: committed offset trails the log end offset; latency grows; in the worst case consumers can't catch up before retention deletes unread data → silent loss. Causes: consumers too slow / too few; a downstream (DB, API) is the bottleneck; partition count caps parallelism; uneven partition load. Fixes: add consumers up to the partition count (then you're capped — repartition to go further); speed up handlers (batch DB writes, async I/O); increase max.poll.records for throughput; scale the slow downstream; alert on lag (not just consumer liveness). If lag is structural, the real fix is more partitions or a faster sink, not more consumers.

9.2 Rebalancing storms#

Symptom: the group constantly rebalances; during (eager) rebalances all consumers stop → throughput collapses; sometimes a livelock loop. Causes: processing exceeds max.poll.interval.ms so the coordinator evicts a "stuck" consumer, whose partitions move, causing more slowness → repeat; frequent pod restarts; aggressive autoscaling; short session.timeout.ms. Fixes: CooperativeStickyAssignor (incremental — only moving partitions pause); static membership (group.instance.id) so restarts don't rebalance; lower max.poll.records or raise max.poll.interval.ms so batches finish in time; stabilize deploys; right-size autoscaling. This is the most common Kafka production pain — know it cold.

9.3 Hot partitions / key skew#

Symptom: one partition (and its consumer) is saturated while others idle; tail latency and lag concentrate on one key. Causes: a partition key with skewed distribution (e.g. a whale tenant, null/default key, low-cardinality key like country). Fixes: choose a higher-cardinality/composite key; for a known hot key, salt it (key + bucket) to spread across sub-partitions — but only if you can relax strict per-key ordering; isolate whales onto their own topic/partitions; reconsider whether you need per-key ordering at all.

9.4 Duplicate processing#

Symptom: the same event handled twice → double charges, double emails. Cause: at-least-once delivery — reprocessing after a crash between processing and offset commit; producer retries without idempotence; outbox relay re-publishing. Fixes: idempotent consumers — dedup on a stable message/business key (DB unique constraint, upsert, Redis seen-set with TTL); make the side effect naturally idempotent; enable idempotent producer to kill producer-path dupes; Kafka transactions for Kafka-to-Kafka EOS. Accept that duplicates will happen and design handlers to tolerate them — "we'll just make sure it's only sent once" is a red flag answer.

9.5 Poison pills (a message that always fails)#

Symptom: one un-processable record (bad schema, malformed, triggers a bug) blocks the whole partition because offsets can't advance past it → head-of-line blocking. Fixes: Dead Letter Queue (DLQ) — after N retries, publish the record to a *.DLT topic with error metadata and commit past it so the partition proceeds; alert and inspect/replay DLQ out of band. Use bounded retries + backoff (Spring Kafka / Kafka Connect have built-in DLQ + retry topics). Guard deserialization (error-handling deserializers) so a single bad byte-sequence doesn't crash the loop. Never silently drop — a DLQ preserves the record for investigation.

9.6 Ordering violations#

Symptom: events for one entity processed out of order (e.g. updated before created). Causes: related events on different partitions (wrong/missing key); multi-threaded consumers reordering within a partition; producer retries without idempotence; a repartition moving a key. Fixes: key by the entity so its events share a partition; keep single-threaded-per-partition processing (or a key-affinity thread pool); enable idempotent producer (preserves order on retry, max.in.flight ≤ 5); make handlers tolerate out-of-order with version/sequence checks (ignore stale updates) where strict ordering isn't feasible.

9.7 Data loss#

Symptom: acknowledged events never make it to consumers. Causes: acks=1 and the leader dies before replication; min.insync.replicas=1 so a lone replica accepts writes then fails; unclean.leader.election=true electing a stale leader; retention deleting unconsumed data; auto-commit committing before processing. Fixes: the durability triangle — acks=all + min.insync.replicas=2 + unclean.leader.election=false + RF≥3; manual commit after processing; retention long enough to absorb consumer downtime; monitor lag vs retention.

9.8 The dual-write inconsistency#

Symptom: DB and Kafka disagree — event published but DB rolled back, or DB committed but event lost. Cause: treating two independent writes as if atomic (§8.3). Fix: Transactional Outbox (+ Debezium CDC). Never publish directly from application code alongside a DB write. Consumers dedup because outbox delivery is at-least-once.

9.9 Backpressure / overload#

Symptom: consumers can't keep up with a downstream; memory bloat; timeouts cascade. Fixes: consumer pause()/resume() to stop fetching when the downstream is saturated; bound in-flight work; batch and bulk-write downstream; let lag act as a natural buffer (Kafka's superpower — the log absorbs bursts so you size for average, not peak). On the producer side, buffer limits + max.block.ms apply backpressure upstream.

9.10 Schema evolution breaks consumers#

Symptom: a producer change deserializes-fails across many consumers at once. Fix: Schema Registry with enforced BACKWARD/FULL compatibility; only additive changes with defaults; breaking change → new topic version; test with old + new schemas in CI. Moves failures to publish time. (§7)

9.11 "Exactly-once" misused for external systems#

Symptom: team enables EOS and assumes the Postgres/API write is now exactly-once. It isn't. Fix: EOS covers Kafka-to-Kafka only. For external sinks: idempotent writes, outbox, or dedup keys. Set expectations explicitly.

9.12 Zombie producers (transactions)#

Symptom: a stalled producer thought dead resumes and writes duplicates/corrupts a transaction. Fix: transactional epoch fencing — a new producer with the same transactional.id bumps the epoch, so the old (zombie) instance is rejected by the broker. Use a stable transactional.id per logical producer.

9.13 Large messages#

Symptom: big payloads blow past message.max.bytes / max.request.size, hurt latency, bloat memory. Fix: claim-check pattern — store the blob in S3/object storage, put a reference (pointer + hash) on Kafka. Kafka is for events, not files. Only raise size limits for modest, bounded cases.

9.14 Rebalance data-in-flight loss/dup during partition revocation#

Symptom: on rebalance, in-flight processed-but-uncommitted records are reprocessed by the new owner (dup) or offsets are committed for records not fully processed (loss). Fix: implement ConsumerRebalanceListener.onPartitionsRevoked to commit finished offsets before releasing partitions; use cooperative rebalancing to minimize churn; keep handlers idempotent so reprocessing is safe.

9.15 Operational / cluster-level#

• Under-replicated partitions: broker/network issues shrink the ISR → durability at risk. Alert on UnderReplicatedPartitions > 0.
• Disk exhaustion: retention misconfig or a lagging log filling disks → broker crash. Monitor disk; set retention deliberately; use tiered storage (KIP-405) to offload old segments to object storage.
• Too many partitions cluster-wide: slows controller failover and recovery. Budget partitions per broker.
• Under-provisioned metadata quorum: in KRaft, size and monitor the controller quorum as you would any consensus group.

10. Architecture & modern context#

• KRaft over ZooKeeper: Kafka replaced ZooKeeper with a self-managed KRaft metadata quorum (Raft-based controllers, KIP-500). Removes a whole operational system, scales to far more partitions, and speeds controller failover. ZooKeeper was fully removed in Kafka 4.0. Expect "why did they drop ZooKeeper?" — answer: single system to operate, faster metadata propagation, higher partition ceilings.
• Tiered storage (KIP-405): offload old log segments to object storage (S3/GCS) while brokers keep recent data on local disk. Decouples storage from compute → cheap, effectively unbounded retention and faster broker recovery (less local data to replicate).
• Kafka Connect: declarative source/sink integration (Debezium CDC, S3, Elasticsearch, JDBC) without writing consumers. Reach for it before hand-rolling glue.
• Kafka Streams / ksqlDB: stream processing inside your app — joins, windowed aggregations, stateful transforms with EOS (exactly_once_v2) and local state stores backed by compacted changelog topics. Flink is the heavier alternative for large/complex topologies.
• Queues / share groups (KIP-932, early access in 4.0): a new "share group" consumer model that adds queue-like semantics — per-message acknowledgement and consumer parallelism beyond the partition count — narrowing Kafka's historical gap vs SQS/RabbitMQ. Worth name-dropping as the direction of travel, but note it's still maturing.
• When NOT to use Kafka (a strong senior answer): request/response RPC; tiny scale where a managed queue (SQS/RabbitMQ) is simpler; strict global ordering needs; workloads wanting priority queues or (historically) per-message ack/visibility-timeout — though share groups are starting to address the latter. Kafka shines at high-throughput, replayable, multi-consumer event streams — not as a general task queue.

11. Observability (what to monitor)#

• Consumer lag per group/partition (the #1 health signal — lag vs retention tells you if you're about to lose data). Tools: Burrow, Kafka Exporter + Prometheus.
• Under-replicated / offline partitions (durability at risk).
• ISR shrink/expand rate, leader-election rate (churn = instability).
• Request latency (produce/fetch p99), broker disk/CPU/network.
• Rebalance frequency & duration per group.
• DLQ volume (rising = a systemic processing problem).
• End-to-end latency (produce timestamp → consume) as a business SLI.

12. Interview framing — how to sound senior#

• Lead with the log model, then derive everything (replay, multi-consumer, decoupling) from it.
• Default to at-least-once + idempotent consumers. Treat "exactly-once everywhere" as a smell; explain EOS's Kafka-only boundary.
• Name the durability triangle unprompted when asked about reliability: acks=all + min.insync.replicas=2 + no unclean election + RF≥3.
• Reach for the outbox pattern the instant a scenario writes to a DB and publishes an event. This is the highest-signal thing you can say.
• Talk trade-offs, not features: partition count (parallelism vs overhead vs repartition pain), ECST vs notification (autonomy vs duplication), choreography vs orchestration (coupling vs visibility), consistency vs availability (min.insync.replicas, unclean election).
• Show you know it's eventually consistent and design for out-of-order, duplicate, and delayed events rather than pretending they won't happen.
• Know when not to use Kafka. Reaching for the right tool is a seniority signal.

One-paragraph "design an event-driven system" template#

Ingest via producers with acks=all, idempotence on, keyed by entity for per-key ordering. Size partitions for peak throughput and consumer parallelism with headroom. Services own local read models fed by ECST events (autonomy over callbacks). Cross-service writes use the outbox + CDC to avoid dual-write; multi-step flows use an orchestrated saga with idempotent compensations. Consumers are at-least-once and idempotent, with DLQs for poison pills and cooperative rebalancing + static membership for stability. Schemas live in a registry with backward compatibility. Durability = RF3 + min.insync.replicas=2 + no unclean election. Monitor lag, under-replicated partitions, and DLQ volume.

13. Config cheat-sheet#

Quick self-test: Can you explain why acks=all alone can still lose data? Why increasing partitions can break ordering? Why exactly-once doesn't cover your database write? What the outbox pattern solves and why it still needs consumer dedup? What actually happens during a rebalance and how cooperative + static membership help? If yes, you're ready.