Trace Context & Context Propagation in Distributed Systems

What this is: a deep, practical guide to what trace context actually is, the wire formats that carry it, how it flows across every hop (HTTP, gRPC, Kafka, async/reactive), and how to reason about it in a system-design interview. Code is shown in Kotlin and Java with Spring Boot 3 (Micrometer Tracing + OpenTelemetry).

The one idea (mental model)#

A trace is a distributed stack trace. Trace context is the pointer you pass across every process boundary so each service knows which trace — and which parent — its work belongs to. "Propagation" is nothing more exotic than copying a few well-known headers, in-band, on every outbound call.

In a monolith, a stack trace is free: the call stack is the causal chain, and a thread-local can hold a correlation ID for the whole request. The moment a request crosses a network boundary, that continuity is gone — the callee gets a fresh thread, a fresh stack, and no idea it is part of your request. Distributed tracing rebuilds the lost call stack by having every service:

1. Extract an incoming context from the request (headers/metadata).
2. Create its own span, whose parent is the incoming span and whose trace ID is inherited unchanged.
3. Inject the (now updated) context into every outbound request it makes.

The trace ID is the thread that survives all hops; each hop swaps in its own span ID as the new "parent" for the next hop. Stitch all the spans that share a trace ID by parent/child and you get the tree. Everything else in this guide is detail on what is in that context, what format it travels in, and where propagation tends to break.

Two things that are constantly conflated — keep them separate:

• Trace context (identity): the minimal state needed to correlate — trace ID, the caller's span ID, sampling decision, and vendor trace state. This is plumbing; it means nothing to your business logic.
• Baggage (application context): arbitrary key/value pairs you choose to carry alongside the trace (tenant ID, user ID, feature flags). This is your data, propagated on the same rails.

1. The data model: traces, spans, and the context that links them#

• Trace — the whole tree of work for one logical request, identified by a trace ID (128-bit / 16 bytes in W3C & OTel).

• Span — one unit of work (an HTTP handler, a DB call, a Kafka publish), identified by a span ID (64-bit / 8 bytes). A span has a start/end time, a name, a status, attributes (key/values), events, and a link to its parent span.

• SpanContext — the immutable, serializable subset of a span that must travel to the next service. This is the "trace context" proper. In OpenTelemetry it is exactly:

  Field	Size	Meaning
  TraceId	16 bytes (32 hex)	Shared by every span in the trace
  SpanId	8 bytes (16 hex)	The current span — becomes the parent of the next hop
  TraceFlags	1 byte	Bit 0 = sampled; (W3C L2) bit 1 = random trace-id
  TraceState	key/value list	Vendor-specific, ordered; carries e.g. sampling probability
  IsRemote	bool (local only)	Was this context extracted from the wire?

• Span link — a non-parent reference to another span, used when one span is caused by many others (e.g. a batch consumer processing N messages, each from a different trace).

The crucial split for propagation:

• What propagates on the wire: TraceId, the caller's SpanId (as parent), TraceFlags, TraceState, and (separately) Baggage. That's it — a few dozen bytes.
• What stays local: span name, attributes, timings, events, status. These are exported straight to your backend (Jaeger, Tempo, Zipkin, an OTLP collector), not passed to the next service.

In-process vs cross-process context#

Propagation has two halves, and interviews love the second:

• Cross-process — serialize context into request headers/metadata and parse it on the other side. This is the W3C/B3/etc. formats.
• In-process — carry the active context from the entry point down to the code that makes the outbound call, so the injector knows what to write. This lives in an ambient container:
  • Blocking code → a thread-local Context (OTel Context, Brave TraceContext), and MDC for logs.
  • Reactive (Project Reactor) → the Reactor Context, not thread-locals.
  • Kotlin coroutines → the coroutine context element.
  • Any hand-off to another thread (thread pool, @Async, CompletableFuture, Kafka listener container) → context must be explicitly captured and restored, or it is silently lost.

The single most common real-world bug: cross-process propagation works, but in-process propagation drops the context at a thread boundary (an executor, a reactive operator, a message listener), so the child work starts a brand-new trace. The trace "breaks" even though every header format was correct.

2. Why it matters#

Without a propagated context, you have per-service logs and per-service latency and no way to connect them. With it, you unlock:

• Correlation / root-cause. One trace ID ties together every log line, span, and error across 20 services. "Request abc was slow" becomes "the 4th of 6 downstream calls, the inventory DB query, took 1.9s of the 2.1s total."
• Critical-path & latency analysis. The span tree shows what ran in parallel vs serially and where the wall-clock time actually went — not guessable from per-service p99s.
• Consistent (coherent) sampling. The sampling decision is made once, at the head, and the sampled bit propagates. Either the whole trace is kept or the whole trace is dropped. Without propagating that bit, service A keeps its spans and service B independently drops its own → you get useless half-traces.
• Service dependency maps & error attribution. Backends derive the live topology and blast radius from parent/child edges.
• Signal correlation (exemplars). A metrics spike links to an exemplar trace; a log line links to its trace. This is only possible if all three signals share the trace/span IDs — which requires propagation into the logging MDC.
• Cross-team debugging. The team that owns service A can see why their call was slow inside service F without access to F's logs.

What breaks without it: no stitching (orphan spans), blindness to cross-service causation, inconsistent sampling (partial traces), and logs that can't be joined. In an incident this is the difference between a 5-minute and a 5-hour diagnosis.

3. Anatomy of W3C Trace Context (the lingua franca)#

W3C Trace Context is the vendor-neutral standard and the default in OpenTelemetry and Spring Boot. Level 1 is a full W3C Recommendation (Feb 2020); Level 2 is a Candidate Recommendation (2024) that mainly adds the random-trace-id flag. It defines two HTTP headers.

traceparent (required, fixed format)#

• version — 00 today. Parsers must be lenient with higher versions (forward compatibility); ff is invalid.
• trace-id — 16 bytes, 32 lowercase hex. All-zero is invalid. Shared across the whole trace.
• parent-id (a.k.a. span-id) — 8 bytes, 16 hex. The caller's span ID. All-zero is invalid. Each hop overwrites this with its own span ID before calling the next service.
• trace-flags — 1 byte. Bit 0 = sampled (01 = sampled, 00 = not). W3C Level 2 defines bit 1 = random (the rightmost 7 bytes of the trace-id are truly random — lets backends sample on the ID without coordination).

A malformed or all-zero traceparent must be treated as absent → the receiver starts a fresh trace (a classic silent "trace reset").

tracestate (optional, vendor multi-value)#

• A comma-separated, ordered list of up to 32 key=value members (the standard budgets ~512 bytes; trim from the right/oldest when over budget).
• Left-most = most recently written. A vendor mutating its own entry moves it to the front.
• Used for state that must survive but isn't identity — most importantly OTel consistent-probability sampling stores its threshold/randomness here (the ot= entry with th:/rv:), so every service can make the same keep/drop decision.
• Unknown keys must be forwarded untouched — never drop another vendor's state.

Interview trap: people say "the trace ID is in tracestate." No — identity is in traceparent; tracestate is vendor-specific extra state (sampling, routing). Getting this backwards is a tell that you've only read a blog post.

4. The types of trace context / propagation formats#

"Types of trace context" has two valid readings — cover both in an interview.

Reading A — the semantic categories of propagated context#

1. Trace identity — trace ID + parent span ID + sampling flag. The non-negotiable minimum (W3C traceparent).
2. Vendor trace state — ordered vendor key/values that must survive hops (W3C tracestate): sampling thresholds, routing hints.
3. Baggage — your arbitrary application key/values (W3C baggage): tenant, user, locale, experiment bucket. Not part of the span; a separate concern that rides the same transport.

Reading B — the wire formats / standards (the ones you actually configure)#

Notes that separate senior from junior answers:

• W3C is the interop default; B3 is what you meet in the wild. Envoy/Istio service meshes historically propagate B3 by default (increasingly W3C-capable). If half your fleet is behind a mesh and half is on OTel defaults, you get format mismatch and broken traces unless you standardize or run a composite (see Problems).
• X-Ray and GCP formats are structurally different — X-Ray's trace ID embeds a timestamp; GCP's span ID is decimal, not hex. You can't just rename headers; you need a real propagator.
• Jaeger, OT Trace, OpenCensus are deprecated in OTel core — supported for migration, not for greenfield.
• Baggage is a separate format from the trace identity, even in W3C. Turning on W3C trace propagation does not automatically propagate baggage; that's a separate propagator you enable.

Transport bindings (the "carrier")#

The same logical context maps onto different carriers:

• HTTP → request headers.
• gRPC → request metadata (Metadata / trailers).
• Messaging (Kafka, RabbitMQ, SQS/SNS) → message/record headers (Kafka Headers, AMQP headers, SQS message attributes). This is where most teams' traces fall apart.
• Binary → the old OpenCensus binary format; effectively dead — prefer text-map everywhere.

5. Propagators: the inject / extract mechanism#

Every format is implemented as a TextMapPropagator with two operations over a carrier (the headers object):

• inject(context, carrier, setter) — write the current context's fields into outbound headers.
• extract(context, carrier, getter) — read headers into a Context you then make current.

The setter/getter decouple the propagator from the carrier type, so the same W3C propagator works on an HTTP header map, a gRPC Metadata, or Kafka Headers — you just supply how to read/write keys on that carrier.

A composite propagator chains several formats: on inject it writes all of them; on extract it tries each until one yields a valid context. This is how you accept B3 and W3C during a migration.

OpenTelemetry defaults to a composite of W3C TraceContext + W3C Baggage. Configuring extra formats (raw OTel SDK, e.g. also emit B3):

Java

Kotlin

With Spring Boot you rarely touch this directly — you set management.tracing.propagation.type and Micrometer wires the propagator for you (next section).

6. How to propagate across every hop (the practical core)#

Spring Boot 3 uses Micrometer Tracing — a facade that bridges to either OpenTelemetry (OTLP export) or OpenZipkin Brave (Zipkin export). It builds on Micrometer Observation: you create an Observation, and tracing + metrics fall out of the same instrumentation. Most of what follows is automatic if you use the auto-configured beans; the job is knowing where automation stops.

6.1 Setup (dependencies + config)#

Dependencies (Gradle, OTel bridge shown; swap the bridge for Brave/Zipkin if you export to Zipkin):

Configuration (language-neutral — this part is identical whatever language your services are in):

Key facts: default sampling is 10%; the default propagation format is W3C; and you can accept multiple formats at once (W3C,B3) during a migration. %X{traceId} reads the MDC that Micrometer populates automatically.

6.2 HTTP — automatic, with one sharp edge#

Inbound extraction and the server span are automatic (a servlet filter / WebFlux WebFilter). Outbound propagation is automatic only if you build clients from the auto-configured builders — this is the #1 gotcha in the Spring docs:

Kotlin

Java

The same rule applies to WebClient.Builder and RestTemplateBuilder: use the injected builder or trace propagation silently won't happen.

6.3 gRPC#

Context rides in gRPC metadata. Use the OpenTelemetry gRPC instrumentation (client & server interceptors from opentelemetry-instrumentation-grpc, or the OTel Java agent) — it injects traceparent into request metadata on the client and extracts it on the server. Manually, it's the same inject/extract as §5 with a Metadata carrier. Don't hand-roll it if the instrumentation is available.

6.4 Messaging (Kafka) — the boundary teams forget#

HTTP is usually fine; async messaging is where traces snap. A producer and consumer are different processes at different times, so the context must be written into the record headers on publish and read back on consume.

With Spring Kafka + Micrometer, turn on observation and it injects/extracts the headers for you:

Kotlin (producer/consumer with observation on — no manual header code needed):

When you are not on Spring Kafka's observation path (raw client, another broker, SQS), do the inject/extract by hand — this is the exact mechanic worth being able to write on a whiteboard:

Java — inject on produce, extract on consume

Kotlin — same mechanic

For batch consumers, one span per record with a span link back to each producer span is the correct model (many causes, no single parent).

6.5 Async, thread pools & reactive — where in-process context leaks#

Cross-process propagation can be perfect and the trace still breaks the instant work hops threads. The context lives in a thread-local; a new thread doesn't have it.

@Async / custom executors — capture the context and restore it on the worker thread via a TaskDecorator using Micrometer's context-propagation library:

Java

Kotlin

Project Reactor — thread-locals don't work across operators; enable automatic bridging once at startup (Reactor 3.5+ / Boot 3):

Kotlin coroutines — the active context must be an element of the CoroutineContext; use the OTel/Micrometer coroutine bridge (e.g. asContextElement()), otherwise a launch/withContext(Dispatchers.IO) starts orphaned work.

Logs — Micrometer copies traceId/spanId into MDC; with the logging pattern from §6.1 every line is greppable by trace ID. If you spawn threads without propagating context, those log lines lose their IDs too — same root cause.

6.6 Baggage — propagate your own fields#

Enable baggage, choose which keys leave the process (remote-fields) and which land in the MDC (correlation):

Kotlin

Java

Use baggage sparingly (see §8) — it is copied onto every downstream request for the life of the trace.

7. Sampling & trace flags in depth#

Sampling is why the flag exists in the context. Two strategies:

• Head-based sampling — decide at the trace's first span whether to keep it, encode that in the sampled bit, and propagate it. Cheap and simple; every service honors the head's decision (ParentBased sampler) so the trace is coherent — all-or-nothing. The downside: you must decide before you know if the request errored or was slow, so you may drop exactly the interesting traces.
• Tail-based sampling — buffer all spans (usually in a collector) until the trace completes, then keep the interesting ones (errors, high latency) and drop the boring 99%. Catches the traces you care about, but requires buffering complete traces (memory + a collector that sees all spans of a trace — routing by trace ID) and adds latency/cost.

Consistent probability sampling is the advanced answer: instead of each service flipping its own coin, a randomness value + threshold travels in tracestate (ot= with rv/th), so independent services reach the same keep/drop decision without a head coordinator — coherent traces even with probabilistic sampling. W3C Level 2's random flag exists to make this reliable.

Interview-level takeaways: the sampled bit must propagate or your sampling is incoherent; head-based = coherent-but-blind, tail-based = smart-but-heavy; production default is often "head-based low % + tail-based for errors/latency in the collector."

8. Baggage in depth (and its dangers)#

Baggage is arbitrary user-defined key/values propagated across the whole trace. It solves the problem: "service F needs the tenant-id that only the edge service knew, and threading it through every API signature is impractical." Put it in baggage at the edge; read it anywhere downstream.

Where it differs from tracestate: tracestate is for tracing-system state (vendors, sampling); baggage is for application data. Different headers, different purpose.

The cautions — a senior candidate volunteers these unprompted:

• It's copied on every hop. N services × M outbound calls each carry the payload. Big baggage = bandwidth + latency tax on the entire trace.
• It leaves your trust boundary. Baggage headers are forwarded to whatever you call next — including third-party APIs — unless you strip them at the egress. Never put PII or secrets in baggage. Whitelist exactly which fields propagate remotely (remote-fields), don't propagate everything.
• Header size limits. Proxies and servers cap total header size; unbounded baggage → requests rejected with 431 / silently truncated.
• It's not free-form storage. Keep it to a handful of small, stable keys (tenant, request priority, experiment bucket), not a scratchpad.

Problems & how to solve them#

Problem: the trace "breaks" at a service boundary — the callee starts a brand-new trace ID. Root cause: the callee never extracted the context (no server instrumentation, or a proxy/framework stripped the headers), so it saw no traceparent and started fresh. Fix: ensure server-side instrumentation is active on the callee; verify traceparent actually arrives on the wire (log inbound headers once); check no gateway/WAF strips non-allowlisted headers.

Problem: two services trace fine alone but don't stitch together. Root cause: format mismatch — one emits W3C traceparent, the other only reads B3 (classic when half the fleet is behind an Istio/Envoy mesh defaulting to B3). Fix: standardize on one format org-wide, or configure a composite that accepts both (management.tracing.propagation.type: W3C,B3) at least during migration; align the mesh and the apps.

Problem: trace ID resets across Kafka / a queue. Root cause: context was never written into message headers on publish (or not read on consume) — HTTP was instrumented, messaging was not. Fix: enable observation-enabled on KafkaTemplate and listener containers, or manually inject on produce and extract on consume (§6.4).

Problem: trace breaks the moment work moves to @Async, a thread pool, CompletableFuture, or a reactive operator. Root cause: in-process context lives in a thread-local; the new thread doesn't inherit it. Fix: TaskDecorator with a Micrometer ContextSnapshot for executors; Hooks.enableAutomaticContextPropagation() for Reactor; the coroutine context bridge for Kotlin.

Problem: logs have no trace/span IDs, so you can't join logs to traces. Root cause: tracer not populating MDC, or a custom log pattern that omits %X{traceId}, or logging from a thread that lost context. Fix: keep the Micrometer MDC bridge on and include %X{traceId}/%X{spanId} in the pattern; fix the underlying thread-context loss (same root cause as above).

Problem: partial traces — some services' spans are kept, others dropped, for the same request. Root cause: the sampled bit isn't being propagated, so each service samples independently. Fix: use a ParentBased sampler that honors the incoming flag; make the head decision authoritative; verify the flag survives every hop (including messaging).

Problem: tracestate or baggage grows without bound; requests start failing with 431 / headers truncated. Root cause: something appends entries every hop (or oversized baggage) and nothing trims to the 32-member / ~512-byte budget or the server header cap. Fix: cap and prune; restrict remote-fields to a tiny allowlist; keep values short.

Problem: tenant-id from baggage shows up in a third party's logs. Root cause: baggage is forwarded on every outbound call, including external ones, and wasn't stripped at egress. Fix: strip baggage headers at the egress gateway/client for external destinations; never carry PII/secrets in baggage; explicit remote-field allowlist.

Problem: an all-zero or malformed traceparent silently starts a new trace. Root cause: an upstream (buggy client, misconfigured LB) sends 00-000…-000…-00; spec says treat as absent. Fix: find and fix the emitter; optionally reject/repair at the edge; alert on a spike of root spans with no parent.

Interview framing / how to sound senior#

Lead with the mental model, not the header syntax: "Trace context is the request's identity carried in-band across every hop; propagation is extract-on-the-way-in, inject-on-the-way-out. The trace ID is constant, each hop becomes the parent of the next."

Then hit the points that signal depth:

• Name the standard and why it exists: W3C Trace Context (traceparent/tracestate) is the vendor-neutral default; B3 is the older Zipkin format you still meet everywhere (meshes). OTel defaults to W3C + Baggage.
• Separate identity from baggage from vendor state — three different things people conflate. Identity in traceparent; vendor/sampling state in tracestate; your app data in baggage.
• Volunteer the real failure mode: "HTTP propagation is usually automatic; the traces break at async boundaries — Kafka, thread pools, reactive — because in-process context is a thread-local that doesn't cross threads." This is the single most senior thing you can say here.
• Sampling must be coherent: the sampled bit propagates so a trace is kept or dropped as a whole; head-based (cheap, blind) vs tail-based (smart, heavy); consistent probability sampling via tracestate for the best of both.
• Cost & security of baggage: it's copied on every hop and crosses trust boundaries — allowlist fields, no PII, mind header-size caps.
• Operational stance: standardize on one format org-wide (or run a composite at the edge during migration); prefer auto-instrumentation (OTel agent / Micrometer) over hand-rolled inject/extract, but be able to write the inject/extract by hand.

Common wrong answers / traps to avoid:

• "The trace ID lives in tracestate." (No — traceparent.)
• "Turning on tracing propagates baggage too." (No — separate propagator.)
• "We generate a new trace ID in each service." (That defeats the purpose — you inherit the trace ID and only mint a new span ID.)
• Ignoring messaging and thread hand-offs — describing only HTTP.
• Forgetting the sampled bit, so traces come out half-sampled.

Cheat-sheet#

W3C traceparent fields

Format quick-reference

Spring Boot knobs

Do / don't

• Do: inherit trace ID, mint new span ID, propagate the sampled bit, use auto-configured client builders, instrument messaging + thread hand-offs, allowlist baggage.
• Don't: mint new trace IDs per service, put PII in baggage, hand-build HTTP clients, assume HTTP-only, mix formats without a composite.

Self-test#

1. What exactly travels on the wire as "trace context," and what stays local to the service?
2. Draw a valid traceparent and label every field with its byte/hex length. Which values are invalid?
3. What's the difference between traceparent, tracestate, and baggage? Which one holds the trace ID?
4. A service receives a request, calls two downstream services, and publishes a Kafka event. Describe exactly what it extracts, what it creates, and what it injects — and how the IDs change.
5. Your HTTP traces are perfect but everything after a Kafka topic shows up as separate traces. Diagnose and fix.
6. Why can a trace break when work moves onto an @Async thread even though nothing about the network changed? How do you fix it in Spring?
7. Two teams both emit valid traces but their spans never join into one tree. Give the most likely cause and two fixes.
8. Explain head-based vs tail-based sampling and why the sampled bit must be propagated. What does consistent probability sampling add?
9. Name three risks of putting data in baggage and how you mitigate each.
10. Which propagation format is the OTel/Spring Boot default, which is Envoy/Istio's historical default, and how do you make a mixed fleet interoperate?

Lineage & sources: W3C Trace Context Level 1 (Recommendation, 2020) and Level 2 (Candidate Recommendation, 2024) — traceparent/tracestate; W3C Baggage; OpenTelemetry Propagators API (W3C TraceContext + Baggage default composite; B3 core; Jaeger/OT/OpenCensus deprecated); OpenZipkin B3 propagation; Spring Boot 3 Actuator "Tracing" docs + Micrometer Tracing (bridges to OpenTelemetry and OpenZipkin Brave). Version-dependent facts (spec statuses, Spring property names/defaults) verified July 2026 — re-check the Spring Boot reference for your exact minor version, since propagation produce/consume and Zipkin-bridge deprecations move between releases.