System design, step by step · Lesson 5 of 7 · 14 min

Reliability Patterns Explained: Retries to Circuit Breakers

Reliability patterns explained: SLOs, error budgets, redundancy, failover, retries with backoff and jitter, bulkheads, backpressure, circuit breakers.

By Matthews WongReviewed on 13 October 2026

Key takeaway

Reliability patterns are the habits that let a system fail well: when a part breaks, the damage stays small and short. You set an SLO below 100% and spend the gap as an error budget, run redundant copies behind health checks, retry at one layer with jittered backoff, cap each dependency with bulkheads and backpressure, and trip a circuit breaker when a dependency keeps failing.

You will be able to

Turn an SLO into an error budget in minutes, and judge what redundancy really buys.
Explain why layered retries become a storm, and fix it with one retry layer, jittered backoff and a budget.
Walk a circuit breaker through closed, open and half-open, and say when to reach for it.

Watch it work

Step through the diagram

Press Play, or step with the arrows. Every step is written out below the diagram.

From a retry storm to a circuit breaker

A payment provider starts failing; retries at two layers multiply into 9 calls per click, retrying at one layer with jittered backoff spreads them out, and a circuit breaker trips open after repeated failures, half-opens after a timeout to let trial calls through, and closes when they succeed.

A client app calls the orders API, which calls a payment provider. Both layers retry a failed call up to 3 times. The circuit breaker is not added yet.

Keys: ← → step · Home / End · K play or pause

A client app calls the orders API, which calls a payment provider. Both layers retry a failed call up to 3 times. The circuit breaker is not added yet.
The provider starts answering 503: it is overloaded. That looks transient, so both layers retry it, exactly as their code says.
Predict the next step Both layers retry 3 times. How many calls reach the provider per click? (3 calls · 6 calls · 9 calls)
Retries multiply: 3 tries at the app times 3 at the API is 9 calls per click, and every client fires them together. A short outage becomes a storm.
Fix one: only the API retries, after a capped exponential wait with full jitter, a random pause up to 200, then 400 ms. Clients stop retrying in lockstep.
sleep = random(0, min(cap, base * 2 ** n))
Fix two: a circuit breaker counts failures. At the threshold it trips open, and calls fail at once without reaching the provider, which gets room to recover.
@failure_threshold = 5 # Fowler's examplePredict the next step The breaker is open. What happens when its timeout runs out? (It closes and sends all traffic · A few trial calls get through · It stays open until restarted)
When the open timeout ends, the breaker goes half-open and lets a limited number of trial calls through. If any of them fails, it trips straight back to open.
The trial calls succeed, so the breaker closes and resets its failure count. Normal traffic flows again, and jittered retries still absorb the occasional blip.

How reliable does a system need to be, and what is an error budget?

Start with a number, not a feeling. A service level indicator (SLI) is a ratio of good events to valid events, such as successful requests over all requests. A service level objective (SLO) is the target for that ratio over a window, and the Google SRE Workbook is blunt that 100% is the wrong target: users cannot tell the difference, and chasing it slows every release. An SLA is the contract with consequences, and it should be looser than the SLO so your own alarm rings first.

What do redundancy and failover really buy you?

Parts in a request path are in series, so their availabilities multiply: three tiers at 99.9% each give 0.999 cubed, about 99.7%. Redundant copies of one tier are in parallel, so the tier is down only when every copy is: two independent 99.9% copies give 1 minus 0.001 squared, 99.9999% on paper. The words on paper matter. That formula assumes the copies fail independently and that failover is instant, and two replicas on one host, one config push or one deploy pipeline fail together.

How do you retry without starting a retry storm?

A retry is a bet that the next attempt will succeed, and it is also extra load aimed at something that is already struggling. Retry only transient failures, such as timeouts, connection resets, 429 and 503, and only operations that are safe to repeat. RFC 9110 calls a method idempotent when sending it twice has the same intended effect as sending it once. A POST is not, so a retried payment needs an idempotency key created before the first attempt and reused by every retry.

How do bulkheads, backpressure and circuit breakers contain a failure?

A bulkhead gives each dependency its own fixed share of a finite resource, a connection pool, a set of concurrency slots or a container's memory, so a slow dependency can exhaust only its own share. Azure names the pattern after the sections of a ship's hull. Size each share with Little's law: concurrent calls equal arrival rate times latency, so 40 requests a second held for 0.05 seconds need 2 connections. Backpressure applies the same idea between a producer and a consumer: bound every queue and decide what happens when it is full, and at the edge answer 429 or 503 with a Retry-After header saying how long to wait.

The real config behind this lesson

Excerpts from the repository that builds and deploys this website, quoted as they are.

docker-compose.yml:27-42

    mem_limit: 512m
    cpus: "1.0"
    healthcheck:
      test:
        [
          "CMD",
          "wget",
          "--no-verbose",
          "--tries=1",
          "--spider",
          "http://127.0.0.1:3000/robots.txt",
        ]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

This site runs one container, so the honest redundancy figure is zero: if the box or the container is down, the site is down. Two other ideas from this lesson are here. mem_limit and cpus are a container-level bulkhead: the app may use at most 512 MB and one CPU, so a runaway render cannot starve the other containers on the same VPS. The health check is detection only: three failed checks 30 seconds apart mark the container unhealthy, but the restart: unless-stopped policy on line 10 reacts to the process exiting, not to that label.

app/api/chat/route.ts:392-397

        // Groq's rate limit (org-wide tokens/requests per minute) throws a 429
        // before streaming starts. Surface it as its own code so the client can
        // say "busy, try again shortly" instead of "connection interrupted".
        // Duck-typed on status so it doesn't depend on the SDK's error class.
        const status = (error as { status?: unknown } | null)?.status;
        const code = status === 429 ? "rate_limited" : "stream_failed";

The chat route adds no retry loop of its own when the model provider answers 429: it reports busy and lets the visitor decide, which is the SRE book's advice to let an overload error bubble up. One layer below sits a retry the route does not show. The client is created with new Groq({ apiKey }), and groq-sdk retries 408, 409, 429 and 5xx errors 2 times by default with a short exponential backoff. So a rate-limited request reaches the provider up to 3 times before this branch runs, against a quota every visitor shares.

lib/api/errors.ts:125-133

export function rateLimited(limit: number, resetIn: number): Response {
  return apiError({
    status: 429,
    code: "rate_limited",
    message: "Too many requests. Please try again later.",
    hint: `Wait ${resetIn} seconds before retrying; the limit is ${limit} requests per minute per client.`,
    retryAfter: resetIn,
    headers: {
      "Retry-After": String(resetIn),

Load shedding at this site's edge. When a client goes over its per-minute limit, the API answers 429 at once with a Retry-After header, instead of queueing work it cannot afford. The counts behind it live in memory in the one running container (lib/rate-limit.ts). That works only because there is one copy: a second replica would give every client a second allowance, which is why shared state has to move out before a tier can be made redundant.

What breaks

Mistakes that cost real time

Retries at every layer
An HTTP client, an SDK, a queue consumer and a gateway can each retry by default, and nobody sees the product until the dependency falls over. List every layer between the user and the dependency and decide which single one retries; turn the others off.
Backoff without jitter
Doubling the wait spaces retries out over time, but if every client fails at once they all wait the same 200, 400 and 800 ms and hit the dependency in waves. Randomise each wait with full jitter, and keep the cap, because jitter does not bound the delay.
Retrying a POST without an idempotency key
After a timeout you cannot tell whether the server did the work. Retrying a payment then charges twice, unless the client created one idempotency key before the first attempt and the server returns the stored result for a key it has already seen.
Redundancy that fails together
Two replicas behind one load balancer, on one host, fed by one deploy pipeline, are one failure with two names. Look for the shared parts, the proxy, the host, the config push, before counting the copies.
A breaker tuned for the wrong recovery time
Azure warns about both ends: a breaker that stays open too long keeps failing calls after the fault is fixed, and one that half-opens too quickly flaps and slows every caller. Match the open timeout to how the dependency actually recovers, and alert when a breaker opens.
What I'd change here: an invisible retry layer
This site's chat route creates its Groq client without setting maxRetries, so the SDK's default of 2 retries applies to every rate-limited request. Passing maxRetries explicitly, 0 to fail fast or 2 with a comment, would make the retry decision visible in the code instead of in a dependency.

Retry, circuit breaker or bulkhead: the question each one answers
	Retry	Circuit breaker	Bulkhead
Question	Does this call deserve another try?	Should I call this dependency at all?	How much capacity may it take?
Trigger	A transient error on one call	Failures past a threshold	A fixed limit on slots or memory
State it keeps	An attempt count per call	Closed, open or half-open	Calls in flight
Goes wrong when	Every layer does it, in lockstep	The timeout ignores real recovery time	Shares add up past the real limit

Try it yourself

Do it on your own machine

You need Node 22 or later and nothing else. A tiny server fails for its first 5 seconds; three clients hit it in different ways, and the server prints how many requests it received each second.

Save the flaky server as server.mjs and start it in its own terminal. Restart it before each of the next steps, so the 5-second outage starts again.

// server.mjs: answers 503 for the first 5 seconds, then 200, and counts requests
import http from "node:http";
const start = Date.now();
let hits = 0;
http
  .createServer((req, res) => {
    hits += 1;
    const down = Date.now() - start < 5000;
    res.writeHead(down ? 503 : 200).end(down ? "busy\n" : "ok\n");
  })
  .listen(8080, () => console.log("listening on :8080"));
setInterval(() => console.log("requests so far:", hits), 1000);

Cause a retry storm: 50 clients, two layers that each try 3 times with no wait. The server count jumps to 450 within a second, 50 clients times 9.

// storm.mjs: 50 clients, two layers that each try 3 times with no wait
const call = () => fetch("http://127.0.0.1:8080/").then((res) => res.ok);
async function api() {
  for (let i = 0; i < 3; i++) if (await call()) return true;
  return false;
}
async function app() {
  for (let i = 0; i < 3; i++) if (await api()) return true;
  return false;
}
await Promise.all(Array.from({ length: 50 }, app));
console.log("done");

Retry at one layer with capped exponential backoff and full jitter. The count now climbs over several seconds instead of jumping, and how many clients succeed depends on whether their random waits outlast the outage.

// jitter.mjs: one retry layer, capped exponential backoff with full jitter
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
const call = () => fetch("http://127.0.0.1:8080/").then((res) => res.ok);
async function withRetry(attempts = 6, base = 200, cap = 4000) {
  for (let n = 0; n < attempts; n++) {
    if (await call()) return true;
    if (n < attempts - 1) await sleep(Math.random() * Math.min(cap, base * 2 ** n));
  }
  return false;
}
const results = await Promise.all(Array.from({ length: 50 }, () => withRetry()));
console.log("succeeded:", results.filter(Boolean).length, "of 50");

Put a circuit breaker in front. It opens after 5 failures, fails fast while open, half-opens after 2 seconds, and closes once a trial call succeeds. Watch the transitions, and how few requests the server gets during the outage.

// breaker.mjs: closed, open after 5 failures, half-open after 2 s, closed again
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
class Breaker {
  state = "closed";
  failures = 0;
  openedAt = 0;
  constructor(threshold = 5, openMs = 2000) {
    this.threshold = threshold;
    this.openMs = openMs;
  }
  async call(fn) {
    if (this.state === "open") {
      if (Date.now() - this.openedAt < this.openMs) return "failed fast";
      this.move("half-open");
    }
    const ok = await fn();
    if (ok) {
      this.failures = 0;
      if (this.state !== "closed") this.move("closed");
      return "ok";
    }
    this.failures += 1;
    if (this.state === "half-open" || this.failures >= this.threshold) {
      this.openedAt = Date.now();
      this.move("open");
    }
    return "failed";
  }
  move(next) {
    console.log(this.state, "->", next);
    this.state = next;
  }
}
const breaker = new Breaker();
const call = () => fetch("http://127.0.0.1:8080/").then((res) => res.ok);
for (let i = 0; i < 40; i++) {
  console.log(i, await breaker.call(call));
  await sleep(250);
}

Quick quiz

Three questions before you move on

Pick an answer to see why it is right or wrong. Nothing is saved or graded, so guess freely.

1.A thousand clients fail at the same moment and all retry after exactly 200, 400 and 800 ms. What does the recovering provider see?
2.After a deploy, the container's health check fails three times in a row. The service has restart: unless-stopped. What does Docker do?
3.A half-open circuit breaker sends two trial calls. The first succeeds and the second fails. What does the breaker do next?

Think it through

Answer in your head, then check

1.Your SLO is 99.9% over 30 days. How much failure can you afford, and what should happen when it is gone?Show the answer
Answer:0.1% of 43,200 minutes, which is 43.2 minutes. When it is spent, the error budget policy applies: freeze risky changes and put reliability work ahead of features until the service is back within its SLO.
2.Why doesn't a pair of 99.9% servers really give you 99.9999%?Show the answer
Answer:The formula assumes the two fail independently and that failover is instant. A shared host, proxy, config or deploy pipeline makes them fail together, and the time to detect a failure counts as downtime too.
3.Your app and the API behind it both retry 3 times. Where should the retry live?Show the answer
Answer:At one layer only, the one immediately above the dependency that is failing, with jittered backoff and a retry budget. Turn the other layer's retries off, or every failing click becomes 9 calls.
4.When does a circuit breaker help more than a retry?Show the answer
Answer:When the fault is not transient. A retry assumes the next attempt will probably work; a breaker notices that it keeps not working, stops sending calls so the dependency can recover, and tests it again with a few trial calls.

Read the module

Every post in this module, in order

The lesson above is the map. Each post goes deep on one part of it, with the full worked examples.

Sources

Primary sources checked on the research date. Follow them for the current version.

Previous lessonConsistency in Distributed Systems Explained: ACID, CAP, Sagas Next lessonMicroservices for Beginners: Boundaries, Sync vs Async, Queues

Step through the diagram

From a retry storm to a circuit breaker

How reliable does a system need to be, and what is an error budget?

What do redundancy and failover really buy you?

How do you retry without starting a retry storm?

How do bulkheads, backpressure and circuit breakers contain a failure?

The real config behind this lesson

Mistakes that cost real time

Retries at every layer

Backoff without jitter

Retrying a POST without an idempotency key

Redundancy that fails together

A breaker tuned for the wrong recovery time

What I'd change here: an invisible retry layer

Do it on your own machine

Three questions before you move on

Answer in your head, then check

Every post in this module, in order

Sources