Skip to main content

Event-Driven Batch Processing

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 12 — rewritten and condensed for teaching.

Work queues shine at one input → one output. Many batch jobs need multi-step flows or many outputs from one input. Link queues so one’s completion becomes the next’s event — a workflow: work flowing through a directed acyclic graph (DAG) of stages.

flowchart TB
  S1[Stage 1]
  S2a[Stage 2a]
  S2b[Stage 2b]
  S3[Stage 3]
  S1 -->|copier| S2a
  S1 -->|copier| S2b
  S2a --> S3
  S2b --> S3

Without a blueprint of how queues relate, event-driven batch systems become as hard to reason about as sprawling FaaS graphs — especially once errors and retries appear.

Patterns of Event-Driven Processing​

Simple chaining (queue A → queue B) is obvious. The patterns below coordinate multiple queues or reshape streams.

Copier​

Duplicate one stream into two or more identical streams when several independent jobs need the same input.

Example: one uploaded video → 4K, 1080p, mobile, and GIF thumbnail queues.

flowchart LR
  IN[Video]
  C[Copier]
  Q1[4K]
  Q2[1080p]
  Q3[Mobile]
  Q4[GIF]
  IN --> C
  C --> Q1
  C --> Q2
  C --> Q3
  C --> Q4

Also useful as a temporary reliability/latency hack: process twice and take the first success to dodge rare 60× slowdowns or raise effective reliability (e.g. ~99% → ~99.99% if failures are independent) — better to fix the root cause later, but resources can buy time.

Filter​

Shrink a stream by dropping items that fail a predicate.

Example: new signups → keep only users who opted into promo email.

Compose as an ambassador/adapter around an existing source: full list in, filtered list out to the generic queue infrastructure.

Splitter​

Like a filter, but route every item somewhere — no silent drops. Evaluate criteria; send items to different queues.

Example: shipped orders → email-notification queue and/or SMS queue (both if the user chose both).

A splitter can include copy behavior. You could build XOR-style routing from a copier + filters, but a splitter names the intent more clearly (like having an XOR gate even though AND/OR/NOT suffice).

Sharder​

A more general splitter: divide work with a shard function into roughly even queues.

Why:

  • Reliability / rollouts — bad worker image hits one shard first (e.g. 1 of 4 → ~25% blast radius).
  • Load spreading — balance across regions/DCs.

When a shard is unhealthy, routing should spill work to remaining healthy shards (including down to one).

Merger​

Opposite of a copier: many sources → one queue.

Example: many git repos each produce commit events; merge into one build-and-test queue instead of N parallel CI fleets.

Often a multi-adapter: several source containers adapted into one merged ItemList for a shared worker pool.

Hands On: Route with filter, splitter, and sharder​

Check: workflow patterns​

Quiz

Batch workflow patterns

1 / 3

How does a splitter differ from a filter?

Pick an answer to see if you are right.

Hands On: New User Signup Workflow​

Example funnel:

  1. Verify email — shard users across failure zones; each shard sends verification mail; stage ends.
  2. On verify event — copy into welcome-email queue and notification-setup queue.
  3. Welcome email queue sends mail and exits.
  4. Notification path filters/splits into email and/or SMS registration queues.
flowchart TB
  SIGNUP[New signup]
  SHARD[Sharded verify-email queues]
  VERIFY[User clicks verify]
  COPY[Copier]
  WELCOME[Welcome email]
  NOTIFY[Notification setup]
  FE[Email notify]
  FS[SMS notify]
  SIGNUP --> SHARD
  VERIFY --> COPY
  COPY --> WELCOME
  COPY --> NOTIFY
  NOTIFY --> FE
  NOTIFY --> FS

Hands On: Signup stage planner​

Publisher/Subscriber Infrastructure​

Wiring stages with local directories does not scale past one node; network filesystems add ops pain. Prefer a pub/sub service:

  • Topics/queues hold messages.
  • Publishers write; subscribers consume reliably.
  • Each workflow edge is often a topic (e.g. sharder outputs Photos-1 … Photos-3).

Cloud options include Azure Event Grid, AWS SQS, etc.; Kafka is a common self-hosted choice on Kubernetes (often via Helm). Create topics with sensible replication factor (3 or 5) and partitions (upper bound on consumer parallelism).

Resiliency and Performance​

Real queues mix fast and slow items and face crashes. Design for both.

Work stealing​

When one worker’s queue is empty, steal from the back of the longest peer queue — shortens the worst line (grocery-store lane change) without racing the head item another worker may already own.

Errors, priority, and retry​

Typical queue semantics:

  1. Claim message (hidden from others, not deleted).
  2. Complete to delete permanently.
  3. On timeout / crash → message returns for another worker.

Implications:

  • Assume at-least-once → make handlers idempotent (or use locks/ownership).
  • Poison pills that crash 100% of workers can cascade — track attempt counts and use exponential backoff, eventually quarantine.
  • After a widespread outage, a huge retry backlog delays fresh user work — use a dual queue: never-tried vs retry (priority: drain new work alongside retries), generalizing to multi-priority queues for latency goals.

Hands On: Work steal and poison backoff​

Check: resiliency​

Quiz

Work stealing and retries

1 / 3

Why steal from the *back* of another worker’s queue?

Pick an answer to see if you are right.

Key takeaways
  • Event-driven batch workflows chain queues into a DAG — blueprint the graph or ops becomes opaque.
  • Core patterns: copier, filter, splitter, sharder, merger (compose freely).
  • Wire stages with pub/sub (Kafka, cloud queues) rather than brittle shared folders.
  • Plan for skew (work stealing), crashes (claim/complete, idempotency), poison (backoff), and backlogs (priority / dual queues).

Further Reading​