Skip to main content

Unifying Batch and Stream Processing

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 12 — rewritten and condensed for teaching.

Data Integration and the Limits of Total Order established the goal: make sure data ends up in the right form in all the right places. Batch and stream processors (Chapter 10 and Chapter 11) are the tools for actually doing that — consuming inputs, transforming, joining, filtering, aggregating, and writing to the right outputs. This lesson looks at what the two approaches share, and how the industry has moved from running them side by side toward genuinely unifying them.

Maintaining Derived State​

Batch and stream processing share a strong functional flavor, whether or not the code itself is written in a functional language: deterministic, pure functions whose output depends only on their input, with no side effects beyond their explicit output, treating input as immutable and output as append-only. Stream processing extends this with one addition — managed, fault-tolerant operator state (see Rebuilding state after a failure).

This discipline pays off twice over: it's what makes idempotent retries safe, and it makes the dataflow across an entire organization dramatically easier to reason about — search index, statistical model, or cache, it's all just a pipeline deriving one thing from another.

Derived data could, in principle, be maintained synchronously — a relational database updates its secondary indexes inside the very same transaction as the write to the indexed table. But asynchrony is exactly what makes event-log-based systems robust: a fault in one component stays local, instead of the way distributed transactions abort everywhere the instant any single participant fails (see Limitations of distributed transactions). This is also why partitioned secondary indexes — which routinely have to cross partition boundaries — are most reliable when maintained asynchronously rather than synchronously.

Reprocessing Data for Application Evolution​

Stream processing keeps derived views current with low delay; batch processing lets you reprocess the entire accumulated history to derive a genuinely new view. That second capability matters more than it sounds — without it, schema evolution is limited to additive, low-risk changes (a new optional field, a new record type). With reprocessing, you can restructure a dataset into a fundamentally different model to meet new requirements.

Schema Migrations on Railways

19th-century England had competing railway gauges (the distance between rails) — trains built for one gauge simply couldn't run on another. The eventual fix, after standardizing in 1846, wasn't shutting a line down for a rebuild: engineers added a third rail to create dual gauge track. Both old and new trains ran on the same line for years, until every train had converted, at which point the now-redundant rail could finally be removed.

The same pattern works for datasets: maintain the old schema and the new schema side by side, as two independently-derived views of the same underlying data. Route a small fraction of users to the new view, watch for bugs, gradually increase that fraction, and eventually retire the old view. Every stage of that process is easily reversible — you always have a working system to fall back to — which is exactly what lets you move faster with real confidence.

The Lambda Architecture​

If batch reprocesses history and stream processing handles what just happened, the natural question is how to combine them. The lambda architecture proposes running two systems in parallel against the same immutable, append-only event log: a batch system (e.g., Hadoop MapReduce) produces a slower, exact, "corrected" view, while a stream processor (e.g., Storm) produces a fast, approximate view in the meantime. The premise: batch processing is simpler and less bug-prone, while stream processors were, at the time, considered less reliable and harder to make properly fault-tolerant.

Despite being genuinely influential — it popularized deriving views from immutable event streams and reprocessing when needed — Kleppmann identifies real practical problems with running it as literally specified:

  • Duplicated logic. The same computation has to be implemented and maintained in both a batch and a streaming framework — libraries like Summingbird help express one computation that can run in either, but the operational burden of debugging and tuning two separate systems doesn't go away.
  • Merging two outputs. The batch and stream pipelines produce genuinely separate outputs that must be merged to answer a user request — trivial for a simple tumbling-window aggregation, much harder for joins, sessionization, or non-time-series output.
  • Expensive full reprocessing. Reprocessing the entire history on every batch run is costly at scale, which pushes the batch layer toward incremental batches instead — reintroducing exactly the straggler and window-boundary problems from stream time semantics, and eroding the "keep the batch layer simple" goal that motivated the split in the first place.

Unifying Batch and Stream Processing​

More recent systems get the lambda architecture's benefits without its duplication, by implementing both modes in one engine. That requires three capabilities, now increasingly standard:

  1. Replaying historical events through the same engine that handles the live stream — log-based brokers already support replay (see Replaying old messages), and some stream processors can also read directly from a distributed filesystem like HDFS.
  2. Exactly-once semantics for stream processing — the output matches what it would have been with no faults at all, which (as in batch processing) requires discarding any failed task's partial output.
  3. Event-time windowing, not processing-time windowing — processing time is meaningless once you're reprocessing history (see Reasoning About Time). Apache Beam's API, runnable on Flink or Google Cloud Dataflow, is a concrete example of expressing computations this way.

Practice: reconciling a lambda-architecture merge​

Implement the merge step a lambda-style pipeline needs: a fast stream view gives an immediate, approximate answer, and a slower batch view later supplies the corrected value for whatever it's had time to (re)process.

Check: reprocessing, the lambda architecture, and unification​

Quiz

Reprocessing for evolution, the lambda architecture, and unifying the two models

1 / 3

Why does the ability to reprocess an entire historical dataset (not just apply incremental updates) matter for evolving an application's schema?

Pick an answer to see if you are right.

Key takeaways
  • Batch and stream processing share a deterministic, side-effect-free "functional" core, and both should generally be maintained asynchronously — synchronous, transactional maintenance of derived data amplifies local faults into system-wide ones, exactly like distributed transactions do.
  • Batch processing's ability to reprocess an entire historical dataset — not just apply incremental updates — is what makes large, reversible schema and application evolution possible, following the same "run old and new side by side" pattern railways used to change gauges.
  • The lambda architecture popularized deriving views from immutable event logs, but running duplicated logic in two frameworks and merging their separate outputs is real, ongoing operational cost.
  • Modern engines unify batch and stream processing by supporting historical replay, exactly-once semantics, and event-time windowing in one system — capturing the lambda architecture's benefits without its duplication.

Further Reading​