Skip to main content

Problems with Replication Lag

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 5 — rewritten and condensed for teaching.

Leaders and Followers covered how leader-based replication moves data around. This lesson covers what happens once that movement isn't instantaneous — which, in an asynchronously replicated system, it never quite is.

Read-heavy workloads (the common case on the web) tempt you toward a read-scaling architecture: add followers, spread read traffic across them, and only send writes to the leader. It works — but only with asynchronous replication. Force every follower to be synchronous and a single flaky node (or network hiccup) would stall writes across the whole system; the more followers you add, the more likely exactly that happens at any given moment.

The cost of going asynchronous: an application reading from a follower that's fallen behind can see outdated data. Run the identical query against the leader and a lagging follower at the same instant and you may get two different answers. This is temporary — stop writing, wait, and followers eventually converge — which is exactly why it's called eventual consistency. "Eventually" is deliberately vague: under normal conditions the replication lag might be a few milliseconds, unnoticeable in practice; under load or network trouble it can stretch to seconds or, in bad cases, minutes.

When that lag gets large enough to actually matter, it stops being a theoretical footnote and starts breaking real user-facing behavior in three specific, well-studied ways.

Reading Your Own Writes​

Picture a user submitting something — a profile update, a comment — then immediately viewing it. If the write goes to the leader but the view is served from a lagging follower, the user's own just-submitted data can appear to have vanished. That's a bad, confusing experience for exactly the person who just did something.

The fix is a guarantee called read-after-write consistency (or read-your-writes consistency): you will always see updates you just made, on your next read — with zero promise about how quickly other users' updates propagate to you. A few real techniques for implementing it:

  • Route by ownership. If you know a certain kind of data can only be edited by its owner (e.g., a user's own profile), always read that specific data from the leader, and read everything else from followers.
  • Route by recency. Track the time of an entity's last write; for some short window afterward (a minute, say), route reads for it to the leader regardless of who's asking. Alternatively, track each follower's lag directly and skip any follower lagging more than that same threshold.
  • Route by timestamp. Have the client remember the (logical or wall-clock) timestamp of its own most recent write, and require whichever replica serves its next read to be at least that far caught up — waiting briefly, or being redirected elsewhere, if it isn't.
  • Route by datacenter. If replicas span multiple datacenters, any request that must reach the leader has to be routed to whichever datacenter the leader currently lives in.

Multi-device access compounds this: a user editing on their phone, then immediately checking on a laptop, expects to see the same update on both — but the two devices might not even route through the same datacenter, and neither device inherently knows what the other one just wrote. That kind of cross-device guarantee generally needs the "last write timestamp" tracked centrally, not per-device.

Check: read-after-write consistency​

Quiz

Read-after-write consistency

1 / 2

What does read-after-write (read-your-writes) consistency specifically guarantee?

Pick an answer to see if you are right.

Monotonic Reads​

A second anomaly: a user's own reads appearing to move backward in time. Imagine refreshing a page, where each request happens to land on a different, randomly-chosen replica. The first request hits a barely-lagging follower and shows a comment that was just posted; a moment later, a second request lands on a more-lagging follower and the same comment is simply gone. Nothing was actually lost — but from the user's perspective, something they just saw has inexplicably vanished, which reads as much more alarming than if they'd simply never seen it in the first place.

Monotonic reads is the guarantee that rules this out. It's weaker than full consistency, but stronger than plain eventual consistency: you might still see stale data, but if you make several reads in sequence, you will never see time run backward relative to your own prior read. The standard implementation: route each user consistently to the same replica — commonly by hashing something like their user ID — while different users can still land on different replicas. The one wrinkle: if that specific replica goes down, the user's next request needs to be rerouted somewhere else, and the guarantee only holds again once it's re-anchored.

Practice: sticky replica routing​

Implement the core mechanic behind monotonic reads: a deterministic function that always sends the same user to the same replica, so long as that replica is healthy.

Check: monotonic reads​

Quiz

Monotonic reads

1 / 2

What specific anomaly does the monotonic reads guarantee prevent?

Pick an answer to see if you are right.

Consistent Prefix Reads​

The third anomaly is about causality — one write logically depending on another. The book's own memorable example: an observer overhears a question-and-answer exchange, but through replicas with uneven lag — the answer replicates fast, the question replicates slow. What the observer sees, played back in the wrong order, looks like someone answering a question before it was ever asked.

Consistent prefix reads is the guarantee that rules this out: if a sequence of writes happens in a particular order, anyone reading them back will observe that same order — never a shuffled one. This bites hardest in partitioned (sharded) databases, since different partitions typically replicate independently, with no shared global ordering across them — a reader can easily see one partition further ahead in time than another. Two real mitigations: keep any writes that are causally related to each other on the same partition (not always practical), or use an algorithm that explicitly tracks causal dependencies between writes across partitions — a deeper technique this series returns to later.

Check: consistent prefix reads​

Quiz

Consistent prefix reads and causality

1 / 2

What does the "question answered before it was asked" scenario illustrate about replication lag?

Pick an answer to see if you are right.

Living With Eventual Consistency​

None of this means eventual consistency is a mistake — it means it has to be a deliberate choice, not an accident. Before shipping on top of asynchronous replication, it's worth explicitly asking: what does this application actually do if replication lag balloons to minutes, or hours? If the honest answer is "nothing breaks," great — ship it as-is. If the honest answer is "users get confused or lose trust," that's a sign the application needs one of the stronger guarantees covered above (or a stronger guarantee still), not a system quietly pretending to be synchronous while actually running asynchronous underneath.

Handling all of this explicitly, in application code, on every relevant query, is exactly the kind of thing that's easy to get subtly wrong. This is a big part of why transactions exist in the first place — they let the database absorb this complexity so the application doesn't have to reason about it manually, a topic this series returns to directly in a later lesson.

Key takeaways
  • Eventual consistency is what you get from asynchronous replication — replicas converge eventually, with no fixed bound on how long "eventually" might take under load or network trouble.
  • Read-after-write consistency guarantees you see your own recent writes — achieved by routing certain reads to the leader, based on ownership, recency, tracked timestamps, or datacenter.
  • Monotonic reads guarantees your own sequential reads never appear to go backward in time — typically achieved by routing each user consistently to the same replica.
  • Consistent prefix reads guarantees causally related writes are never observed out of order — a particular risk in partitioned databases, where different partitions can drift out of sync with each other independently.
  • These guarantees exist precisely so applications don't have to hand-roll lag-handling logic on every query — which is also a large part of why transactions exist as a concept at all.

Further Reading​