Skip to main content

Common Failure Patterns

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 16 — rewritten and condensed for teaching.

Earlier chapters covered patterns to build. This one is about patterns to avoid — recurring mistakes in design and operation. Knowing what fails (and why) is as useful as knowing what to copy.

The Thundering Herd​

Capacity is finite. When demand (or retries) exceeds it, users and clients hit reload / retry. Those retries double load: new traffic X plus the failed X becomes ~2X, then worse — especially with chained RPCs and nested retry loops.

Mitigations on the client:

  1. Exponential backoff — wait longer after each failure (double the delay).
  2. Jitter — randomize waits so everyone does not wake up together (machines sync; humans are already noisy).
  3. Circuit breaker — trip when error rates cross a threshold; stop sending for a cool-down.
  4. Gradual recovery — ramp traffic back in; dumping 100% onto a recovering service often restarts the herd.
flowchart LR
  FAIL[Failures]
  RETRY[Blind retries]
  MORE[More failures]
  FAIL --> RETRY --> MORE --> RETRY

Check: thundering herd​

Quiz

Retry amplification

1 / 2

Why does blind retry under overload often make an outage irrecoverable?

Pick an answer to see if you are right.

Hands On: Backoff with jitter​

The Absence of Errors Is an Error​

Alerting only on high error rates misses the opposite: zero traffic / zero errors because nothing is happening.

Example: a subtitle pipeline that only processes uploaded movies. If the uploader loses write permission, uploads stop → processing stops → error rate 0% — and a “errors > 10%” alert never fires. The service is still broken.

Alert on too many errors and on too few errors / requests (traffic drop).

Check: silent failure​

Quiz

Missing traffic

1 / 1

Why can a 0% error rate still mean a severe outage?

Pick an answer to see if you are right.

“Client” and “Expected” Errors​

Invalid client input (bad auth, malformed body) should not count against your SLO the same way as 500s. But “client error” buckets become dumping grounds: if all users suddenly get 401, that is usually your auth bug — not “users being wrong.”

Also run synthetic probes you fully control. For those, there should be no expected client errors. Pair real-traffic SLOs with synthetic success checks so misclassified errors still page.

Check: error buckets​

Quiz

Client vs system errors

1 / 1

What risk comes from ignoring all 401/403 as “client errors”?

Pick an answer to see if you are right.

Hands On: Split error classes + traffic floors​

Versioning Errors​

Clients need a stable external API. Internally you want freedom to change memory layout and storage. Keep three versions plus translators:

LayerRole
ExternalClient-facing wire/API versions (many at once)
InternalIn-memory / domain model
StorageOn-disk / DB schema

Coupling them forces lockstep client + server + DB rollouts — not a decoupled distributed system. Separate versions so you can roll forward/back code without rewriting every client or every stored row at once.

Check: API versions​

Quiz

External vs internal vs storage

1 / 1

Why maintain separate external, internal, and storage representations?

Pick an answer to see if you are right.

The Myth of Optional Components​

Caches (and similar “optional” pieces) start as performance extras. Under growth they become required for SLOs — while ops still treats them as disposable. Local in-process caches are especially sneaky: easy to add, hard to size, bad at sharing memory across replicas, and they proliferate.

Treat a new cache like any other permanent architecture change: capacity, reliability, and failure modes included.

Check: optional caches​

Quiz

Optional in name only

1 / 1

What usually happens to an “optional” cache as load grows?

Pick an answer to see if you are right.

Oops, We “Cleaned Up” Everything​

Cross-component failure modes delete state at scale. Classic story: GC deletes photos when the user “doesn’t exist.” User API maps DB outage → 404. GC believes every owner is gone → deletes everything. Backups save the data; trust takes longer.

Humans will mis-map errors across team boundaries. Defense is systemic:

  • Map dependency failures to 5xx, not “not found”
  • Throttle destructive ops (deletes/sec caps)
  • Circuit-break GC when delete rates spike orders of magnitude above baseline until a human resets

Design for the world where calls return the wrong answer, not only the happy path.

flowchart TB
  DB[(Database down)]
  API[User API returns 404]
  GC[GC deletes all photos]
  DB --> API --> GC

Check: runaway cleanup​

Quiz

Catastrophic deletes

1 / 1

Why is returning 404 when the user DB is down dangerous next to a “delete if user missing” GC?

Pick an answer to see if you are right.

Hands On: GC circuit breaker​

Challenges with the Breadth of Inputs​

Some bugs hit 100% for a tiny slice of users — “compiler” problems: infinite legal input space, rare combinations, or fuzzed/malicious payloads (classic SQL injection as the old web example).

Coverage strategies:

  1. Record representative real inputs — enough time window, regions (e.g. CJK text), and time-of-day variation.
  2. Fuzz — random (or LLM-generated) valid-shaped inputs for richer APIs.

Together: don’t break yesterday’s users; harden against weird and hostile tomorrow.

Check: input breadth​

Quiz

Rare but sticky bugs

1 / 1

Why is recording only North American daytime traffic a weak regression corpus?

Pick an answer to see if you are right.

Processing Obsolete Work​

After an outage, a backlog of abandoned work (users already reloaded) can keep latency high for hours even after the root cause is fixed — especially if the system runs near capacity.

Recover faster with:

  1. Timeouts — abort work past a deadline (alone is not enough if each item is just under the timeout).
  2. Scale on latency — autoscale (or manually surge) on user-facing lag, not only CPU.
  3. Prefer newer work — triage: drop or deprioritize stale queue items so fresh reloads succeed.

Short outages with fast recovery beat long “fixed but still slow” tails.

Check: obsolete work​

Quiz

Backlog after recovery

1 / 1

Why can a 10-minute outage cause hours of high latency afterward?

Pick an answer to see if you are right.

Hands On: Prefer fresh work​

The “Second System” Problem​

Seeing debt everywhere tempts a greenfield rewrite. New systems feel easier and more fun — until “ready” never arrives.

Why rewrites stall:

  • Feature catch-up — production keeps shipping; the rewrite trails by months.
  • Mileage debt — bugs in the new system get it turned off; it never earns production hours.

Better: improve one microservice at a time, and invest in boundaries (e.g. storage behind an interface) so you can run two implementations side by side without replacing the world.

Check: second system​

Quiz

Rewrite traps

1 / 1

Why do full “second systems” often fail to replace production?

Pick an answer to see if you are right.

Key takeaways
  • Thundering herd — backoff + jitter + circuit breakers + gradual recovery; never blind retry storms.
  • Alert on high and low traffic/errors; don’t hide regressions in “client” buckets; use synthetics.
  • Separate external / internal / storage versions; treat “optional” caches as permanent.
  • Guard destructive automation with throttle + breaker; shed obsolete work after outages.
  • Prefer incremental service/abstraction rewrites over doomed second systems.

Further Reading​