Circuit Breaker & Bulkhead · Backend Study Library

Circuit Breaker#

A circuit breaker wraps calls to something that can fail and tracks failure rate. Three states:

Closed — normal operation, requests flow, failures are counted.
Open — failure threshold exceeded; requests fail immediately without hitting the target, giving it time to recover.
Half-Open — after a cooldown, a few probe requests are let through; success closes the breaker, failure reopens it.

Its two jobs: fail fast instead of hanging, and stop hammering a struggling system so it can recover — preventing cascading failure.

As API consumer (the classic case)#

You wrap every outbound call (via Resilience4j, Polly, or a mesh like Istio/Envoy):

Why: a slow dependency is worse than a dead one. Without a breaker, calls pile up waiting on timeouts, exhausting your threads/connections — and now you are down too. The breaker converts slow failures into instant failures.
How: configure failure-rate threshold (e.g. 50% over a sliding window), slow-call threshold (calls slower than X count as failures), open duration, half-open probe count. In Resilience4j:

java
CircuitBreakerConfig config = CircuitBreakerConfig.custom() .slidingWindowType(SlidingWindowType.COUNT_BASED) .slidingWindowSize(20) // judge over the last 20 calls .minimumNumberOfCalls(10) // no verdict before 10 calls .failureRateThreshold(50) // ≥50% failures → OPEN .slowCallDurationThreshold(Duration.ofSeconds(2)) // calls >2s count as slow .slowCallRateThreshold(50) // ≥50% slow calls → OPEN .waitDurationInOpenState(Duration.ofSeconds(30)) // stay open 30s .permittedNumberOfCallsInHalfOpenState(5) // 5 probe calls in half-open .automaticTransitionFromOpenToHalfOpenEnabled(true) .recordExceptions(IOException.class, TimeoutException.class) .ignoreExceptions(BusinessException.class) // don't trip on business errors .build(); CircuitBreaker cb = CircuitBreakerRegistry.of(config).circuitBreaker("productService");

Or in Spring Boot application.yml:

yaml
resilience4j.circuitbreaker: instances: productService: slidingWindowSize: 20 minimumNumberOfCalls: 10 failureRateThreshold: 50 slowCallDurationThreshold: 2s slowCallRateThreshold: 50 waitDurationInOpenState: 30s permittedNumberOfCallsInHalfOpenState: 5 ignoreExceptions: - com.example.BusinessException
Fallbacks: when open, return cached/stale data, a default value, a degraded response, or queue for later — decided per use case. The fallback design is the hard part, not the breaker.
Granularity: one breaker per dependency (or per endpoint/host), never one global breaker — otherwise one bad dependency blocks healthy ones.

Fallbacks in Resilience4j

The CircuitBreaker itself has no fallback config — when open it throws CallNotPermittedException; when closed it propagates whatever the call throws. The fallback is your code that catches those and returns a substitute. Two ways:

1. Functional style — decorate the call, then recover:

java

CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("productService");

Supplier<Product> decorated =
    CircuitBreaker.decorateSupplier(cb, () -> productClient.getProduct(id));

Product result = Try.ofSupplier(decorated)          // Vavr Try (or plain try/catch)
    .recover(CallNotPermittedException.class,
             ex -> cache.getStale(id))              // breaker OPEN → fail-fast path
    .recover(Exception.class,
             ex -> Product.defaultProduct())        // real call failed
    .get();

2. Spring Boot annotation style — fallbackMethod:

java

@CircuitBreaker(name = "productService", fallbackMethod = "productFallback")
public Product getProduct(String id) {
    return productClient.getProduct(id);
}

// Breaker is open: no call was even attempted → serve stale cache
private Product productFallback(String id, CallNotPermittedException ex) {
    return cache.getStale(id);
}

// Call was attempted and failed (timeout, 5xx, connection error)
private Product productFallback(String id, Exception ex) {
    return Product.defaultProduct();
}

Rules and gotchas:

The fallback method must live in the same class and have the same signature plus a Throwable as last parameter. With overloads, Resilience4j picks the most specific exception type — that's how you give the open-breaker case a different fallback than an ordinary failure.
Distinguishing CallNotPermittedException matters: breaker open means don't bother degrading gracefully per-request logic that itself calls the dependency — the dependency wasn't even tried, so serve the cheapest safe answer.
When combining annotations (@Retry, @TimeLimiter, @Bulkhead on the same method), the fallback fires after the whole decorated chain gives up — e.g. after retries are exhausted, not per attempt.
Keep fallbacks cheap and reliable (in-memory cache, static default). A fallback that makes another network call is a new failure point.

As API publisher#

Strictly, a circuit breaker protects a caller, so on the publisher side it shows up differently:

Protecting your own downstreams: your API calls a DB, cache, other services — you're a consumer of those, so you put breakers on each of them so one sick dependency doesn't take your whole API down.
Self-protection is load shedding / admission control, not CB: when overloaded, reject early with 503 + Retry-After instead of queueing until everything times out. Rate limiting and per-client quotas serve the same "fail fast" role.
Help your consumers' breakers work: return clear, fast error codes (503 for overload, not slow 200s or hung connections), expose health endpoints, don't accept requests you can't serve. A slow, ambiguous failure defeats the consumers' breakers.
At the gateway/mesh: Envoy "outlier detection" ejects unhealthy upstream instances — circuit breaking per instance, done in infrastructure instead of code.

Bulkhead#

Named after ship compartments: partition resources so one flooding compartment doesn't sink the ship. Technically: separate thread pools, connection pools, semaphores (max concurrent calls), or entire deployments per dependency/tenant/workload.

As API consumer#

Give each downstream dependency its own thread pool or connection pool (or a semaphore cap on concurrent calls). If service B goes slow, only B's pool saturates — calls to A and C continue normally. Without this, one slow dependency consumes every worker thread and your whole service stalls.
Semaphore bulkheads (just a concurrency cap) are cheap and usually sufficient; dedicated thread pools add isolation plus timeout control but cost context switching.

Semaphore bulkhead, in detail: it's nothing more than an atomic counter with a limit. Each call acquires a permit before executing and releases it after. The call runs on the caller's own thread — no handoff, no extra pool. If all permits are taken, the incoming call either waits up to maxWaitDuration or fails instantly with BulkheadFullException (set wait to 0 for pure fail-fast). That rejection is the whole point: excess load is turned away at the door in microseconds instead of piling up.

java
BulkheadConfig config = BulkheadConfig.custom() .maxConcurrentCalls(25) // at most 25 in-flight calls to this dependency .maxWaitDuration(Duration.ZERO) // 26th call fails immediately, no queuing .build(); Bulkhead bulkhead = Bulkhead.of("productService", config); Supplier<Product> guarded = Bulkhead.decorateSupplier(bulkhead, () -> productClient.getProduct(id)); // throws BulkheadFullException when saturated → handle like a fallback case

Trade-offs vs. a thread-pool bulkhead (ThreadPoolBulkhead: dedicated fixed pool + bounded queue, caller gets a CompletableFuture):
- Semaphore: near-zero overhead, keeps ThreadLocal context (MDC, security), works for sync and async. But it cannot forcibly time out a stuck call — the caller's thread is still inside the call, so you must pair it with client/socket timeouts.
- Thread pool: hard isolation (a hung dependency can only consume its own pool) and real timeouts when combined with TimeLimiter, at the cost of context switching, lost ThreadLocals, and more tuning (pool size, queue capacity).
- Rule of thumb: default to semaphore + proper client timeouts; reach for thread-pool when the dependency is slow/untrusted and you need guaranteed cut-off.

As API publisher#

Isolate workload classes: separate worker pools/instances for critical vs. bulk endpoints (e.g. checkout vs. reporting), so an expensive report query can't starve payments.
Tenant isolation: per-client concurrency limits and quotas so one noisy customer can't consume all capacity; dedicated capacity or cells for premium tenants (cell-based architecture is bulkheading at deployment level).
Resource partitioning: separate DB connection pools per endpoint class; separate queues; even separate clusters.
Envoy's "circuit breakers" (max connections/pending requests per upstream) are actually bulkhead-style limits — worth knowing the naming trap.

Interview hints for a tech lead#

State the distinction crisply: circuit breaker = fail fast + give recovery time (reacts to failure over time); bulkhead = contain blast radius (static isolation, works even before failures are detected). They complement each other, not substitute.
They're part of a resilience stack, order matters: timeouts → retries (with exponential backoff + jitter) → circuit breaker → bulkhead → rate limiting/load shedding. Classic Resilience4j decoration order: Retry(CircuitBreaker(RateLimiter(TimeLimiter(Bulkhead(call))))). A breaker without timeouts is nearly useless — slow calls never trip it unless you count slow calls as failures.

What jitter is: randomness added to each retry delay. With plain exponential backoff, every client that failed at the same instant retries at the same instants (1s, 2s, 4s…) — synchronized waves that slam the recovering service again (thundering herd). Jitter spreads them out. Best-known scheme is AWS "full jitter": sleep = random(0, base × 2^attempt) — e.g. attempt 3 with base 1s sleeps a random value between 0 and 8s instead of exactly 8s. Costs nothing, and without it, backoff barely helps at scale.
Retry storms: retries without a breaker amplify outages (N callers × M retries). This cause-and-effect story is a strong senior signal.
Half-open pitfalls: thundering herd when the breaker closes and queued demand slams a barely-recovered service; mitigate with limited probes and gradual ramp-up.
Fallback design > breaker config. "The breaker opened — then what?" Degraded UX, stale cache, default values, queue-and-retry. Tie it to business impact.
Granularity and tuning: per-dependency/per-host breakers; thresholds derived from SLOs and baseline error rates, not guessed. Too sensitive = flapping; too lax = useless.
Library vs. infrastructure: in-code (Resilience4j/Polly) gives fallback logic and app context; service mesh (Istio/Envoy) gives language-agnostic, centrally managed policies but no business-aware fallback. A tech lead should articulate this trade-off and often use both.
Only count the right failures: 4xx/business errors shouldn't trip the breaker — only timeouts, 5xx, connection errors. Common misconfiguration.
Observability & testing: emit metrics/alerts on breaker state transitions (an open breaker is an incident signal), and validate with chaos engineering/fault injection — untested resilience config is fiction.
Know the history: Hystrix (Netflix) popularized both patterns, now in maintenance — say Resilience4j/Polly/mesh instead. Referencing Michael Nygard's Release It! (origin of both patterns) lands well.