Skip to main content

Work Queue Systems

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 11 — rewritten and condensed for teaching.

The simplest batch pattern is a work queue: a pile of independent items, each processable without talking to the others. The usual goal is finishing each item within a latency bound by scaling workers up or down with demand.

flowchart LR
  SRC[Work source]
  Q[Queue / manager]
  W1[Worker]
  W2[Worker]
  W3[Worker]
  SRC --> Q
  Q --> W1
  Q --> W2
  Q --> W3

Work queues showcase distributed system patterns: most of the machinery is generic; only “what is an item?” and “how do I process one?” are app-specific. With containers, those two pieces become reusable library containers plus a thin user worker.

A Generic Container Work Queue​

Split the system into:

RoleResponsibility
SourceStream of work item names/IDs
ManagerTrack pending vs done; spawn workers
WorkerProcess one item’s payload

The source often sits beside the manager as an ambassador (localhost proxy): the generic manager asks a stable API; the ambassador maps that to NFS listings, cloud objects, Kafka, Redis, etc.

flowchart TB
  subgraph group [Container group]
    MGR[Generic work-queue manager]
    AMB[Source ambassador]
  end
  REAL[Real source: NFS / S3 / Kafka / ...]
  MGR -->|localhost API| AMB
  AMB --> REAL

Reuse library ambassadors when the backend is generic; only swap which ambassador you deploy.

Source Container Interface​

Prefer a small versioned HTTP API on localhost (versioning is cheap on day one, expensive to retrofit):

  • GET http://localhost/api/v1/items → list of item names
  • GET http://localhost/api/v1/items/<item-name> → item details

List response shape:

{
  "kind": "ItemList",
  "apiVersion": "v1",
  "items": ["item-1", "item-2"]
}

Item response:

{
  "kind": "Item",
  "apiVersion": "v1",
  "data": { "some": "json", "object": "here" }
}

There is no “mark complete” on this API on purpose: keep completion tracking in the generic manager (maximizes shared logic). The manager passes item.data into the worker interface.

Hands On: Source list endpoint​

Implement the list handler for files matching a suffix (video thumbnailer style).

Check: source API​

Quiz

Source ambassador API

1 / 2

Why version the work-source API as /api/v1/... even if you “will never change it”?

Pick an answer to see if you are right.

Worker Container Interface​

Workers differ from the source API:

  • One-shot: start work once; no chatty localhost loop.
  • Remote: the manager launches workers via the orchestrator (not the same pod as the manager).
  • Harder to spoof: a cluster peer should not casually inject fake HTTP work.

So use a file-based handoff: set WORK_ITEM_FILE to a path containing item.data (often a Kubernetes ConfigMap mounted as a file). Shell-script workers stay simple — no embedded web server required.

You can still share generic worker images (download from object storage → run a user script → upload results) and only swap the script.

flowchart LR
  MGR[Manager]
  JOB[Job / Pod]
  FILE["WORK_ITEM_FILE"]
  MGR -->|create Job + ConfigMap| JOB
  FILE --> JOB

Hands On: Read WORK_ITEM_FILE​

Shared Work Queue Infrastructure​

Manager loop (conceptually):

  1. Load items from the source API.
  2. Diff against Jobs already created for this queue.
  3. Spawn Jobs for missing items (worker interface).
  4. On successful Job completion, treat the item as done.

Kubernetes Job objects help: run once or until success, survive node loss, and carry annotations naming the work item — so the manager can track state without its own database.

forever:
  items ← GET /items
  jobs  ← list Jobs for this queue
  for each item not in jobs:
    create Job for that item
  sleep

Hands On: Diff pending work​

Example: video thumbnailer​

  • Source: list *.mp4 on shared NFS (ItemList).
  • Worker: ffmpeg -i ${INPUT_FILE} -frames:v 100 thumb.png (or a container that wraps ffmpeg).

Thin adapters + generic queue machinery = short path from idea to running batch jobs.

Check: Kubernetes Jobs​

Quiz

Work queue on Kubernetes

1 / 1

How do Kubernetes Jobs simplify a reusable work-queue manager?

Pick an answer to see if you are right.

Dynamic Scaling of Workers​

Unlimited Job fan-out finishes work fast but bursts cluster load (and cost). Cap parallelism → smoother resource use, higher item latency under load.

If average arrival stays high, a capped queue never catches up — latency grows without bound. Then raise parallelism (or buy capacity).

Stability rule of thumb:

  • Let interarrival = average time between new items (e.g. items / 24h).
  • Let service time = average processing time once work starts (exclude queue wait).
  • With parallelism (p), effective service ≈ service_time / p.
  • Need effective service < interarrival (with safety margin).

Examples:

InterarrivalService (serial)Result
60 s30 sStable; can drain bursts
60 s60 sBalanced but fragile — little slack
60 s120 sUnstable — backlog → ∞

Scale up when effective service approaches interarrival; scale down carefully (e.g. keep service ≈ 90% of interarrival for margin). Prefer battle-tested autoscalers such as KEDA (files, pub/sub, and many other event sources) over reinventing metrics plumbing.

Hands On: Stability check​

Check: scaling math​

Quiz

Work queue scaling

1 / 2

Items arrive every 60 s on average and each takes 120 s alone. With parallelism 3, is the queue stable?

Pick an answer to see if you are right.

The Multiworker Pattern​

One item may need several steps (detect faces → identify → blur). A single megaworker is hard to reuse when the next pipeline wants cars instead of faces but the same blur.

Multiworker (an adapter specialization): one facade container implements the worker interface and delegates to a group of reusable step containers.

flowchart TB
  IFACE[Unified worker interface]
  A[Detect]
  B[Identify]
  C[Blur]
  IFACE --> A
  IFACE --> B
  IFACE --> C

Compose shared steps into new pipelines without rewriting everything.

Check: multiworker​

Quiz

Multiworker composition

1 / 1

What problem does the multiworker pattern solve?

Pick an answer to see if you are right.

Key takeaways
  • Work queues process independent items; scale workers for latency SLAs.
  • Split source ambassador, generic manager, and worker — maximize shared containers.
  • Source: versioned localhost HTTP; worker: WORK_ITEM_FILE (often ConfigMap) for one-shot remote jobs.
  • Managers can track state via Kubernetes Jobs + annotations without custom storage.
  • Keep effective service time < interarrival; use caps and autoscalers (e.g. KEDA) for cost and stability.
  • Multiworker composes reusable step containers behind one worker API.

Further Reading​