Skip to main content

Coordinated Batch Processing

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 13 — rewritten and condensed for teaching.

Event-driven batch showed how to split and chain queues. Many jobs also need to pull parallel outputs back together into an aggregate answer — the other half of patterns like MapReduce.

flowchart TB
  IN[Input set]
  M1[Shard / map]
  M2[Shard / map]
  M3[Shard / map]
  AGG[Join or reduce]
  OUT[Aggregate result]
  IN --> M1
  IN --> M2
  IN --> M3
  M1 --> AGG
  M2 --> AGG
  M3 --> AGG
  AGG --> OUT

Sharding a work queue is the map-like fan-out; this chapter focuses on coordinated stages that wait or merge those results.

Join (Barrier Synchronization)​

Merge blends queues into one stream but does not guarantee a complete dataset before the next stage. You cannot safely compute globals (sums, “all frames ready”) if items still trickle in.

Join (barrier synchronization): parallel work proceeds, but nothing leaves the join until every expected parallel piece finishes.

Example: render every film frame independently, then encode one second of video — you need all 30 frames for that second present first.

flowchart LR
  A[Worker A]
  B[Worker B]
  C[Worker C]
  J[Join / barrier]
  N[Next stage]
  A --> J
  B --> J
  C --> J
  J -->|all done| N
UpsideDownside
Completeness before aggregationLess overlapping parallelism → higher end-to-end latency

Mitigate latency as in earlier chapters: duplicate stragglers / trade compute for time when tail latency dominates.

Check: join vs merge​

Quiz

Join barriers

1 / 2

When do you need a join instead of a simple merge of queues?

Pick an answer to see if you are right.

Hands On: Frame barrier​

Reduce​

If sharding is map-like, reduce is the coordinated merge of many parallel outputs into fewer representatives of the answer — without necessarily blocking until the entire map finishes.

Each reduce step combines several outputs into one (fewer items; often less payload — only fields needed for the final answer). Because reduce output looks like reduce input, you can stack reduce stages until one result remains — and start early while map shards are still running.

Too many reduce layers waste CPU; too few start to look like a single fat join.

Count (word count)​

Shard pages by page-number suffix → each worker emits {word: count} → pairwise (or n-way) reduce sums counts per word → repeat until one dictionary.

Sum​

Same structure for numeric totals (e.g. US population: shard by state → county → town populations → reduce by adding (label, value) pairs). Nested sharding does not change the reduce code if every output is the same shape.

Histogram​

Merge relative histograms by weighting with population (or counts), combining buckets, then renormalizing — again a repeatable reduce.

Check: reduce vs join​

Quiz

Reduce coordination

1 / 2

How does reduce differ from join in a batch pipeline?

Pick an answer to see if you are right.

Hands On: Reduce helpers​

Hands On: An Image Tagging and Processing Pipeline​

Suppose you have many rush-hour highway photos and want vehicle type and color counts, after blurring license plates for privacy. Inputs are HTTPS URLs to raw images.

Stage 1 — blur (sharded multiworker): one container detects plate boxes; another blurs those regions. Group them with the multiworker pattern so each container stays reusable (the blur box API can later anonymize faces in another pipeline). Shard images across queues for throughput and blast-radius control.

Stage 2 — join: upload blurred copies, but do not delete originals until every shard finished — catastrophic failure may need a full rerun. A join holds outputs until all blur shards complete.

Stage 3 — copier: after the barrier, copy each item to:

  • a queue that deletes originals
  • a queue that detects vehicle type and color

Stage 4 — detect (sharded multiworker again): shard detection; pair a “locate vehicle type” worker with a “color this region” worker. Separation lets the color container serve other jobs.

Each image emits JSON like:

{
  "vehicles": { "car": 12, "truck": 7, "motorcycle": 4 },
  "colors": { "white": 8, "black": 3, "blue": 6, "red": 6 }
}

Stage 5 — reduce: tree-reduce those partials (same idea as word count) until one aggregate remains.

flowchart TB
  URLS[Image URLs]
  BLUR[Sharded blur multiworker]
  JOIN[Join all blur shards]
  COPY[Copier]
  DEL[Delete originals]
  DET[Sharded detect multiworker]
  RED[Reduce counts]
  URLS --> BLUR --> JOIN --> COPY
  COPY --> DEL
  COPY --> DET --> RED

Check: pipeline composition​

Quiz

Image pipeline patterns

1 / 2

Why join after sharded blurring before deleting originals?

Pick an answer to see if you are right.

Hands On: Aggregate vehicle stats​

Key takeaways
  • Coordinated batch joins act as barriers — wait for a complete parallel set before the next stage (completeness over overlap).
  • Reduce merges partials into fewer outputs of the same shape; it can start early and stack until one aggregate remains (count, sum, histogram).
  • Real pipelines compose earlier patterns: shard, multiworker, join, copier, then reduce — e.g. blur → barrier → delete+detect → aggregate stats.
  • Join raises latency (stragglers); reduce trades careful associativity for more parallelism.

Further Reading​