Skip to main content

Observing Derived State

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 12 — rewritten and condensed for teaching.

Designing Applications Around Dataflow described a process that keeps derived datasets — search indexes, materialized views, predictive models — continuously up to date. That process has a name: the write path. This lesson looks at its counterpart, the read path, and where pushing that boundary all the way to the end user leads — before this arc closes out with what it actually takes to keep all of this correct, and the ethical questions that come with building it.

Materialized Views and Caching​

The write path is precomputed — done eagerly, the moment new data arrives, whether or not anyone has asked to see it yet. The read path only happens on demand, when a user actually makes a request. Readers familiar with functional programming will recognize the shape: the write path is eager evaluation, the read path is lazy evaluation. A derived dataset is exactly the point where the two meet — trading work done at write time against work done at read time.

Full-text search makes the trade-off concrete. No index at all means zero write-time work, but every search degenerates into scanning every document (like grep) — brutally expensive at scale. Precomputing every possible query's results would eliminate essentially all read-time work, but the space of possible queries is unbounded, so that's not implementable. A real search index sits in between: moderate work at write time (updating index entries for a document's terms) buys moderate work at read time (looking up each query term, then combining results with the query's Boolean logic).

There's a third option, cache-shaped: precompute results for a fixed set of the most common queries — call it a cache, or, just as accurately, a materialized view, since it needs updating whenever new documents affect one of those common queries' results. An index isn't the only place the read/write boundary can sit — full scanning, a search index, and a cache of common queries are three different points along the same trade-off, and this is the exact same idea as the differently-drawn read/write boundary between celebrity and ordinary users introduced all the way back in the book's opening chapter.

Implement both ends of that trade-off directly: serve a query instantly from a precomputed cache (the write path) when one exists, and fall back to scanning the index and combining results with AND logic (the read path) otherwise.

Stateful, Offline-Capable Clients​

Two decades of web application development have made the stateless-client, authoritative-server model feel like the only option — browsers that could do essentially nothing useful offline. But single-page JavaScript apps and mobile apps now routinely hold substantial local state and persistent local storage, enabling offline-first applications that work primarily against a local, on-device database and sync with servers in the background whenever a connection exists. Reframed through the write-path/read-path lens: on-device state is a cache of server state. The pixels on screen are a materialized view of local model objects, which are themselves a local replica of state living in a remote datacenter.

Pushing State Changes to Clients​

Ordinary web pages only reflect a single point-in-time read — a page doesn't find out the underlying data changed until you reload it, effectively polling by hand. Protocols going beyond plain request/response — server-sent events, WebSockets — let a server keep an open connection and actively push changes to the browser as they happen, directly reducing how stale that client-side state can get.

In write-path/read-path terms, pushing state changes all the way to client devices means extending the write path all the way to the end user. A client still needs an initial read to bootstrap its state, but from then on it can simply consume a stream of changes — and it already has a known answer for the "what if the device goes offline for a while" problem: the same consumer offset tracking that lets a log-based broker's consumer resume without missing anything after a disconnect (see Replaying old messages) works identically here, per user, per device.

End-to-End Event Streams​

Frameworks like Elm and Facebook's React/Flux/Redux toolchain already manage client-side UI state as a stream of events — user input, server responses — structured very much like event sourcing. Extending that same model so a server can push state-change events into that same client-side pipeline creates a genuinely end-to-end write path: from the interaction that triggers a change, through event logs and several derived-data stages, all the way to another user's screen — potentially with delay under a second.

Instant messaging and online games already work this way. The obstacle to doing this everywhere is how deeply the request/response, stateless-client assumption is baked into most databases, libraries, and protocols — very few data stores support "give me a stream of future responses" as a first-class operation the way they support ordinary reads and writes. Making that shift — from request/response toward publish/subscribe — is, in Kleppmann's view, worth the real engineering effort it takes, for meaningfully more responsive UIs and genuinely better offline support.

Reads Are Events Too​

Writes to a derived store already flow through an event log; reads are normally a completely separate, transient network request straight to whatever node holds the relevant data. That's a reasonable design — but not the only one. Read requests can be represented as events too, routed through the same stream processor as the writes, with the processor answering a read by emitting its result to an output stream.

Framed this way, a plain one-off read is a stream-table join that immediately forgets the read event once it's answered; a subscription is the same join, kept persistently open against past and future events on the write side. This reframing has a genuine secondary benefit: logging read events lets you reconstruct exactly what state a user actually observed before making a decision — directly enabling the causal-dependency tracking discussed back in Ordering events to capture causality — at the real cost of extra storage and I/O for every read.

Multi-Partition Data Processing​

For a query touching just one partition, routing it through a stream feels like overkill. But the same infrastructure — message routing, partitioning, joining — opens up distributed execution of queries that genuinely span multiple partitions. Storm's distributed RPC feature is a concrete example: computing how many distinct people have seen a given URL on Twitter means unioning the follower sets of everyone who tweeted it — a computation that inherently spans however the user base happens to be partitioned. Fraud prevention hits the same shape: scoring a purchase's risk means joining reputation data for the buyer's IP, email, billing address, and shipping address — each held in its own, separately-partitioned reputation database.

MPP databases' internal query execution graphs already look like this. If you genuinely need this kind of multi-partition join, an off-the-shelf database that already supports it is usually simpler than reimplementing it on a stream processor — but treating queries themselves as streams is a real option once you're building at a scale conventional off-the-shelf tools can no longer comfortably reach.

Check: pushing state to clients, and reads as events​

Quiz

Offline-capable clients, end-to-end streams, and reads as stream-table joins

1 / 3

What does it mean to say that pushing state changes all the way to end-user client devices "extends the write path to the end user"?

Pick an answer to see if you are right.

Key takeaways
  • The write path (eager, precomputed, done regardless of whether anyone asks) and the read path (lazy, done only on demand) meet at every derived dataset — indexes, caches, and materialized views are just different points along that same write-time-vs-read-time trade-off.
  • Offline-first clients reframe on-device state as a cache of server state, and pushing changes to clients via WebSockets/SSE — rather than making clients poll — effectively extends the write path all the way to the end user, using the same consumer-offset machinery that lets a disconnected log consumer resume without missing anything.
  • End-to-end event streams, already used internally by frameworks like Elm and Redux, could in principle extend into a genuinely end-to-end write path — from the interaction that triggers a change all the way to another user's screen — but request/response assumptions remain deeply baked into most current databases and protocols.
  • Representing reads as events, processed through the same stream operator as writes, reframes a one-off read as a transient stream-table join and a subscription as a persistent one — with the added benefit of enabling causal-dependency tracking, at a real storage and I/O cost.
  • Multi-partition query processing (fraud scoring, cross-follower-set unions) can be implemented by routing queries through the same partitioning and joining infrastructure stream processors already provide — a real option once a workload outgrows what an off-the-shelf database comfortably handles.

Further Reading​