Skip to main content

Modes of Dataflow

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 4 — rewritten and condensed for teaching.

Formats for Encoding Data covered how data gets turned into bytes, and how that choice determines whether old and new code can coexist safely. But compatibility is really a relationship between whoever encodes the data and whoever decodes it — and there's more than one way those two parties can be arranged. This lesson covers the three shapes that relationship most commonly takes: through a database, through a service call, and through asynchronous message passing.

Dataflow Through Databases​

The simplest case: the process that writes encodes the data; the process that reads decodes it. If it's the same process at two different points in time, storing something in a database is really just sending a message to your own future self — which makes backward compatibility non-negotiable: your future self has to be able to read what you wrote today.

More realistically, several processes share one database at once — multiple services, or multiple instances of the same service running for scale or fault tolerance. During a rolling upgrade, some of those instances run old code and some run new code, simultaneously, against the same data. That means forward compatibility matters here too — old code has to survive reading records a newer sibling process already wrote.

The Read-Modify-Write Trap​

Here's a scenario worth internalizing: newer code adds a field and writes a value into it. Later, older code (which has no idea that field exists) reads that same record, modifies something unrelated, and writes the whole record back. The field the old code never learned about is now at risk of quietly vanishing — not because the encoding format failed to preserve it (Thrift/Protobuf/Avro all handle unknown fields fine at the wire level), but because the application code in between might not. If a database value gets decoded into an application-level model object and later re-encoded from that same object, any field the model class doesn't know about can get dropped in translation — an application-level bug, not an encoding-format bug, and one that's easy to miss unless you're specifically watching for it.

Data Outlives Code​

A database can hold a five-millisecond-old record right next to a five-year-old one, still sitting in whatever encoding was current when it was written. Deploying new application code might take minutes; migrating an entire dataset to a new schema is usually expensive enough that most systems just... don't, unless forced to. This is often summarized as: data outlives code.

Most relational databases lean into this directly — adding a nullable column doesn't require rewriting existing rows; the database just returns null (or the column's default) for any row that predates the column. Schema evolution is what makes an entire database appear to follow one consistent schema, even when the bytes on disk actually span several historical schema versions.

For archival snapshots — periodic dumps for backup or analytics — there's no reason to preserve that historical mess: since you're copying the data anyway, encode the copy consistently under the latest schema. An immutable, one-shot dump like this is exactly what a format like Avro's object container files (or an analytics-oriented column format) is built for.

Check: dataflow through databases​

Quiz

Databases as a dataflow mechanism

1 / 3

Why does a database inherently require BOTH forward and backward compatibility, once more than one process (or a rolling upgrade) is involved?

Pick an answer to see if you are right.

Dataflow Through Services: REST and RPC​

The second shape: client and server, with the server exposing an API (a service) that clients call over a network. The web itself works this way — browsers are clients, web servers expose HTTP as the API. A server can also be a client to some other service, which is exactly how a large application decomposes into a service-oriented or microservices architecture — the whole point being that each service can be deployed and evolved independently, which means old and new versions of both clients and servers must coexist, which brings us right back to the compatibility question this whole chapter is about.

REST vs. SOAP​

REST isn't a protocol — it's a design philosophy built directly on top of HTTP's own primitives: URLs identify resources, HTTP verbs express intent, and caching/auth/content negotiation reuse HTTP's own built-in machinery. An API that follows these conventions is called RESTful.

SOAP takes the opposite approach: an XML-based protocol that treats HTTP as just one possible transport underneath a large, separately-specified family of standards (collectively "WS-*"). A SOAP API is described in WSDL, which enables strong code generation for statically typed languages — genuinely useful there, considerably less useful for anything dynamically typed, and SOAP tooling in general leans hard on code generation and heavyweight IDE support to be usable at all. Despite formal standardization, cross-vendor SOAP interoperability has historically been a real pain point — one reason it's fallen out of favor outside large enterprises, while REST has become the default, especially for cross-organization integration.

Why RPC Keeps Reinventing Itself — and Keeps Hitting the Same Wall​

Long before REST, a whole lineage of technologies — EJB, Java RMI, DCOM, CORBA — tried to make a network call to another machine look and feel like an ordinary local function call. This idea is called Remote Procedure Call (RPC), and the ambition of "the network should be invisible" is called location transparency. It's a fundamentally leaky abstraction, for reasons that don't go away no matter how good the tooling gets:

Local function callNetwork request
Succeeds or fails predictablyCan be lost, delayed, or fail for reasons entirely outside your control
Returns, throws, or never returnsCan also time out — leaving you with no idea whether it actually happened
Retrying is always safeRetrying a lost response (not a lost request) can silently duplicate the action, unless you build in deduplication
Roughly constant latencyLatency can range from sub-millisecond to many seconds under load
Pass pointers to local memory cheaplyEvery argument must be fully encoded into bytes — fine for small values, awkward for large ones
Same language, same type system on both endsClient and server may use different languages entirely, forcing the RPC framework to reconcile mismatched type systems

None of this means RPC is a bad idea to use — it means it's a bad idea to pretend a network call is a local one. Modern RPC frameworks (gRPC on Protocol Buffers, Thrift's own RPC support, Avro RPC, Finagle) mostly embrace this now rather than hiding it: they expose calls as futures/promises that can explicitly fail, and some (gRPC) support genuine streaming — an ongoing series of requests and responses over time, not a single call-and-response.

Practice: fix the retry bug​

The "retrying a lost response can duplicate an action" problem above is exactly the kind of thing worth feeling in code, not just reading about. Below, charge_customer gets called multiple times for what should be one logical charge (simulating an automatic retry after an ambiguous timeout). Add idempotency using a request ID, so a repeated call with the same ID never charges twice.

Encoding and Evolution for RPC​

RPC gets one simplifying assumption that plain databases don't: it's usually reasonable to assume servers get upgraded before clients do. That narrows the requirement to backward compatibility on requests (the server must handle requests from not-yet-upgraded clients) and forward compatibility on responses (an old client must tolerate responses from an already-upgraded server). Whatever compatibility guarantees the underlying encoding provides — Thrift/gRPC/Avro's tag or name-based rules from the previous lesson — carry straight through to the RPC layer built on top of them.

Compatibility gets genuinely harder across organizational boundaries: you can't force an external client to upgrade on your schedule, so a public API's compatibility promises may effectively need to hold indefinitely — which is exactly why so many public APIs end up maintaining several API versions side by side, using a version number embedded in a URL, an HTTP header, or a stored per-client setting.

Check: services, REST, and RPC​

Quiz

REST, SOAP, and the problems with RPC

1 / 3

What is the core philosophical difference between REST and SOAP?

Pick an answer to see if you are right.

Message-Passing Dataflow​

The third shape sits between the other two: asynchronous message passing, typically through a message broker (a.k.a. message queue). Like RPC, a message reaches its destination with relatively low latency. Like a database, it's not a direct connection — the broker durably stores the message temporarily, decoupling sender from receiver in time as well as space.

Routing messages through a broker buys several things a raw RPC call doesn't:

  • Buffering when a recipient is slow or temporarily unavailable, instead of the call simply failing.
  • Automatic redelivery if a consumer crashes before finishing, preventing silent message loss.
  • No need to know the recipient's address — useful anywhere the actual location of a service instance is fluid, like cloud deployments where machines come and go.
  • Fan-out — one message, many recipients.
  • Logical decoupling — the sender publishes and moves on; it doesn't need to know or care who (if anyone) actually consumes the message.

The trade-off: message passing is normally one-way. A sender fires a message and doesn't wait around for a reply (any actual response typically travels back over a separate channel, if one exists at all) — a genuinely different interaction shape than RPC's call-and-wait.

Concretely, brokers like RabbitMQ, ActiveMQ, and Apache Kafka work by having producers publish to a named queue or topic, with the broker fanning that out to one or more consumers/subscribers. A message broker imposes essentially no data model of its own — a message is just bytes plus some metadata — so the whole compatibility discussion from the previous lesson applies directly: pick an encoding with solid forward/backward compatibility, and publishers and consumers can be deployed, upgraded, and restarted independently, in any order.

Practice: build a minimal message broker​

Implement the fan-out mechanic directly — one topic, potentially many subscribers, and a publish call that delivers to every current subscriber of that topic.

Distributed Actor Frameworks​

The actor model is a concurrency model where independent actors each hold private local state and talk to each other only via asynchronous messages — no shared memory, no locks, no manual thread juggling. Each actor processes one message at a time, so an individual actor's own logic never has to reason about concurrency at all.

Distributed actor frameworks stretch this same idea across machines: the exact same "send a message to an actor" call works whether the recipient is in the same process or on a different continent — if it's remote, the framework transparently encodes the message, ships it over the network, and decodes it on arrival. Location transparency arguably works better here than in RPC, precisely because the actor model already assumes messages can be dropped or delayed even within a single process — remote delivery just makes an already-expected failure mode somewhat more likely, rather than introducing a brand-new one.

A distributed actor framework is, in effect, a message broker and the actor programming model fused into one package — which means the exact same rolling-upgrade compatibility concerns from this whole chapter still apply, since a message can easily be sent from a node already running new code to one still running old code, or vice versa. Three well-known frameworks land in noticeably different places on this:

  • Akka defaults to Java's built-in serialization (no compatibility guarantees at all, by default) but can be reconfigured to use Protocol Buffers, unlocking real rolling-upgrade support.
  • Orleans defaults to a custom encoding with no rolling-upgrade support out of the box — the documented upgrade path is standing up an entirely new cluster and migrating traffic over, though custom serialization plugins are possible.
  • Erlang/OTP makes changing record schemas surprisingly difficult despite the platform's overall reputation for high availability; rolling upgrades are possible, but need real up-front planning (Erlang's newer, JSON-like maps datatype may ease this over time).

Check: message passing and actors​

Quiz

Message brokers and distributed actors

1 / 3

What capability does routing messages through a broker provide that a direct RPC call does not?

Pick an answer to see if you are right.

Key takeaways
  • Databases as a dataflow mechanism require both compatibility directions simultaneously, since old and new code (and old and new data) routinely coexist — and because data outlives code, migrations are the exception, not the default.
  • REST builds on HTTP's own conventions and has become the dominant style for public APIs; SOAP wraps HTTP in a much larger, separately-specified standards stack, leaning heavily on code generation.
  • RPC tries to make a network call look like a local one, but a network call has failure modes (loss, timeouts, unpredictable latency, ambiguous partial failure) a local call never has — modern frameworks (gRPC, Thrift RPC, Avro RPC) increasingly embrace that difference explicitly rather than hiding it, and idempotency keys are the standard fix for the retry-duplication problem specifically.
  • RPC gets a simplifying assumption databases don't: servers usually upgrade before clients, narrowing the requirement to backward-compatible requests and forward-compatible responses — though cross-organization APIs often need to maintain compatibility indefinitely.
  • Message brokers enable durable, decoupled, one-to-many, typically one-way dataflow; distributed actor frameworks fuse a broker with the actor concurrency model, but still inherit every rolling-upgrade compatibility concern from this whole chapter.

Further Reading​