Skip to main content

Designing Applications Around Dataflow

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 12 — rewritten and condensed for teaching.

Unbundling Databases framed composing specialized storage and processing systems as a kind of "database inside-out" — a design pattern, not a new architecture, drawing on ideas from dataflow languages, functional reactive programming, and logic programming that already existed elsewhere. This lesson works through what that pattern actually implies for how application code and state relate to each other.

Application Code as a Derivation Function​

Every derived dataset is produced by some transformation function applied to its source:

  • A secondary index picks out the indexed columns/fields from each row and sorts by them.
  • A full-text search index applies language detection, word segmentation, stemming, spelling correction, and synonym expansion, then builds an inverted index for fast lookup.
  • A machine learning model is derived from training data via feature extraction and statistical analysis; a prediction is then derived from the model plus new input (and hence, indirectly, from the training data too).
  • A cache is often a precomputed aggregation shaped for exactly how a UI will display it — populating it correctly requires knowing what the UI actually references.

CREATE INDEX's derivation function is common enough to be a database built-in. Full-text indexing gets basic support built in for common languages, but sophisticated tuning is domain-specific. Feature engineering for ML is notoriously application-specific. Whenever the derivation function isn't a standard, cookie-cutter one, custom application code is required — and this is precisely where databases have historically struggled: triggers, stored procedures, and user-defined functions exist, but have generally been an afterthought in database design.

Separation of Application Code and State​

In principle, a database could be a deployment environment for arbitrary application code, the way an operating system is. In practice, databases are poorly suited to this: they don't fit modern development needs like dependency management, version control, rolling upgrades, monitoring, or calling out to network services. Cluster-management tools (Kubernetes, Docker, Mesos, YARN) exist specifically to run application code well, by focusing on doing exactly that one thing.

The resulting split — stateless application logic on one side, database-managed state on the other — is now close to universal in web application deployment: any request can be routed to any server, which forgets everything the instant it responds, while the database is the durable, shared, mutable variable both readers and writers interact with over the network.

There's a real limitation baked into that model, though: in most programming languages, you can't subscribe to changes in a mutable variable — only read it periodically. Databases inherited exactly that passive behavior: your only default option for detecting a change is to poll (repeat the same query again later). Subscribing to changes as a first-class database feature (see API support for change streams) is only just beginning to emerge.

Check: derivation functions and the app/state split​

Quiz

Application code as derivation, and separating logic from state

1 / 3

Why have databases historically struggled to support "custom" derivation functions well, even though they support standard ones like secondary indexes natively?

Pick an answer to see if you are right.

Dataflow: Interplay Between State Changes and Application Code​

Thinking in terms of dataflow means renegotiating that database-as-passive-variable relationship: instead of an application manipulating a passive store, application code responds to state changes in one place by triggering state changes elsewhere — the same idea behind treating a database's own change log as a stream to subscribe to, behind message-passing actor systems, and, going back to the 1980s, behind the tuple-spaces model of distributed computation.

Unbundling the database means taking this same idea — a trigger firing, a secondary index updating in step with the table it indexes — and applying it outside the primary database, to caches, search indexes, ML pipelines, and analytics systems, using stream processing and messaging as the mechanism.

Maintaining derived data this way is a genuinely different problem from ordinary asynchronous job execution, which is what most messaging systems were originally built for:

  • Order matters. If several views are derived from the same event log, they need to process events in the same order to remain mutually consistent — many message brokers don't preserve this when redelivering unacknowledged messages, and dual writes are ruled out entirely for the same reason.
  • No message can be lost. Losing even one message permanently desyncs the derived dataset from its source — both message delivery and derived-state updates need to be genuinely reliable, which rules out actor systems that, by default, keep state and messages purely in memory.

Modern stream processors provide exactly this combination of ordering and reliability, at real scale, while still letting application code run as a stream operator doing whatever custom processing built-in database derivation functions don't cover — composed together, Unix-pipe style, into larger systems built entirely around dataflow.

Stream Processors and Services​

The currently dominant style — services communicating over synchronous REST/RPC calls (see Dataflow Through Services: REST and RPC) — buys organizational scalability through loose coupling between teams. Composing stream operators shares that organizational benefit, but the underlying communication mechanism is fundamentally different: one-directional, asynchronous message streams instead of synchronous request/response.

A concrete comparison. Converting a purchase from one currency to another needs the current exchange rate. Two ways to get it:

  1. Microservices approach: query an exchange-rate service or database for the current rate, synchronously, at the moment you need it.
  2. Dataflow approach: subscribe to a stream of exchange-rate updates ahead of time, keep the latest rate in a small local database, and just query that local copy when a purchase needs converting.

The dataflow version replaces a synchronous network call with a local (even same-process) lookup — faster, and more robust to the exchange-rate service being temporarily unavailable. As Kleppmann puts it: the fastest and most reliable network request is no network request at all. In join terms, this has replaced an RPC call with a stream-table join between purchase events and exchange-rate update events — and, like any such join, it's genuinely time-dependent: reprocessing an old purchase event later requires the exchange rate as it was at the original time, not whatever the current rate happens to be.

Practice: a local rate cache fed by a subscribed stream​

Implement the dataflow-style currency conversion directly: maintain a small local table of exchange rates by applying rate-update events in order, then convert each purchase using whatever rate was known as of that purchase — exactly the stream-table join the lesson describes.

Check: dataflow vs. job queues, and stream processors vs. services​

Quiz

Dataflow's stricter requirements, and streams vs. synchronous services

1 / 3

Why is maintaining derived data via stream processing NOT the same problem as ordinary asynchronous job execution, which most message brokers were originally designed for?

Pick an answer to see if you are right.

Key takeaways
  • Every derived dataset — a secondary index, a search index, an ML model, a cache — is the output of some derivation function applied to its source; standard ones are database built-ins, but custom, application-specific ones require real application code, which databases have historically supported only as an afterthought.
  • The now-standard split between stateless application servers and a stateful database works well operationally, but inherits a real limitation: most systems can only poll a database for changes, since subscribing to changes is only just beginning to emerge as a first-class feature.
  • Thinking in terms of dataflow means treating application code as reacting to state changes and triggering further state changes elsewhere — the same pattern already used inside databases for triggers and index maintenance, now applied outside the database via stream processing.
  • Maintaining derived data via streams has stricter requirements than ordinary job queues: consistent ordering across derived views, and zero tolerance for lost messages.
  • Subscribing to a stream and maintaining a local cache, as in the currency-conversion example, can beat synchronous service calls on both speed and robustness — at the cost of having to handle the resulting join's time-dependence correctly.

Further Reading​