Skip to main content

Ownership Election

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 10 — rewritten and condensed for teaching.

Many distributed designs need exactly one active owner for a resource — a scheduler making placement decisions, a shard leader writing, a single worker draining a queue. Leader election (ownership election) is how replicas agree on who that owner is, and how ownership moves when the current leader fails.

sequenceDiagram
  participant R1 as Replica 1
  participant R2 as Replica 2
  participant R3 as Replica 3
  Note over R1: Leader
  R1--xR1: fails
  Note over R3: Becomes leader
  R1->>R1: recovers (follower)
  Note over R3: Remains leader

Ownership is often the hardest and most important piece of a reliable distributed system — get it wrong and you get split brains, double writes, or stuck locks.

Do You Even Need Leader Election?​

The simplest “ownership” is a singleton: one replica. No election — that process owns everything.

In an orchestrator like Kubernetes you still get:

  • Process crash → restart
  • Hang + health check → restart
  • Node failure → reschedule elsewhere

Rough uptime intuition:

FailureTypical recoveryUptime sketch
Daily container crash, ~2 s restartShort blipOn the order of four nines (~2 s/day)
Node death, ~5 min moveLonger gapWorse — but if every node dies daily, you have bigger problems

Deployments hurt singletons more: you cannot run old and new together, so download/start time is downtime. Daily 2-minute rollouts ≈ two nines; hourly deploys can be worse unless you pre-pull images (more complexity — what you hoped to avoid).

Singletons fit async / background work where occasional full outage is acceptable. They are a bad fit for interactive retail, banking, or games: even brief total downtime is highly visible.

Use multi-replica leader election when you need high availability (roughly four+ nines) with continuous serving while one instance is down.

Check: singleton vs election​

Quiz

Singleton ownership

1 / 2

When is a Kubernetes singleton often “good enough” without leader election?

Pick an answer to see if you are right.

The Basics of Leader Election​

Example: service Foo has replicas Foo-1, Foo-2, Foo-3. Object Bar must have one owner at a time — that replica is the leader.

Do not implement Paxos/Raft yourself for production locks — that is like writing mutexes from raw compare-and-swap in assembly: educational, rarely worth it.

Use a consensus-backed key-value store that already runs those algorithms: etcd, ZooKeeper, Consul. They give a reliable replicated store plus primitives — especially compare-and-swap (CAS) and often TTL — to build locks and elections.

Compare-and-swap​

CAS atomically writes nextValue only if the current value equals currentValue (or the key is absent when you allow create). Pseudocode:

compareAndSwap(key, nextValue, currentValue):
  if key missing and currentValue empty → write nextValue, success
  if key missing and currentValue set → error
  if store[key] == currentValue → write nextValue, success
  else → fail (someone else changed it)

With a TTL, an abandoned key expires back to empty so a dead holder cannot block forever.

Hands On: Simulate CAS​

Implement an in-memory CAS used by later lock exercises.

Deploying etcd

Production clusters use multi-node etcd (often via Helm/operators). Locally, running the etcd binary on 127.0.0.1 is enough to explore CAS and leases; Kubernetes itself depends on etcd for cluster state.

Implementing Locks​

Map a mutex onto the KV store: value "0" / unlocked vs "1" / locked (or store the owner name).

  1. Try lock: CAS "1" over "0"; if the key is missing, CAS create "1" with expected empty.
  2. Block: retry until success (prefer watch over sleep-polling when the store supports it).
  3. Unlock: CAS "0" over "1".
  4. TTL: write locks with expiry so a crashed holder cannot wedge the system forever.
  5. Watchdog: assert/crash if you still run critical work after TTL — never assume you hold the lock forever.

The TTL unlock bug​

Without resource versions (or equivalent fencing tokens):

  1. Process-1 locks with TTL t, then stalls past t.
  2. Lock expires; Process-2 acquires.
  3. Process-1 wakes and unlocks — clearing Process-2’s lock.
  4. Process-3 also acquires → two owners.

Fix: remember the version from the successful lock write; unlock (and renew) only with CAS that checks value and version. If TTL expired and someone else wrote, your unlock fails safely.

Hands On: Lock with versioned unlock​

Check: locks and TTL​

Quiz

Distributed locks

1 / 2

Why add a TTL to a distributed lock?

Pick an answer to see if you are right.

Implementing Ownership (Renewable Locks / Leases)​

Sometimes ownership should last for the lifetime of the process (e.g. one active Kubernetes scheduler among several replicas), not a few milliseconds.

A huge TTL (a week) means slow failover after crash. Better: a renewable lock / lease — hold with a short TTL and renew periodically while healthy.

renew(): CAS same owner value with same version, refreshing TTL
loop every ttl/2:
  if not renew(): handleLockLost()  # stop acting as leader

Renew at ttl/2 to reduce accidental expiry from timing jitter. On loss, stop leader work immediately — often easiest in Kubernetes: exit and let the orchestrator restart you as a follower waiting for the lock.

etcd leases: create with TTL (mk if absent), then periodically set the same owner value with TTL to extend.

Hands On: Lease renew loop logic​

Handling Concurrent Data Manipulation​

Even with correct locks, two replicas can briefly both believe they are leader (extreme scheduling delay: TTL expires, new leader elected, old process wakes before handleLockLost).

Mitigations (defense in depth):

  1. Local check before critical work: still locked and now < lockTime + 0.75 * ttl.
  2. Worker-side validation: store owner identity in the KV; workers reject requests from non-owners.
  3. Send resource version / generation with every request; reject if owner or version mismatch — fixes delayed packets after ownership flip-flops (shard1 → shard2 → shard1 while an old request is in flight).
flowchart LR
  LEADER[Claimed leader]
  WORKER[Worker]
  LOCK[Lock store]
  LEADER -->|request + owner + version| WORKER
  WORKER -->|verify| LOCK
  LOCK -->|ok / reject| WORKER

You still choose a consistency/availability trade-off:

ChoiceMeaning
At-most-oncePrefer losing availability briefly over double-applying
At-least-oncePrefer availability; accept rare duplicate processing

Duplicates may only waste GPU time — or may double-charge a bank transfer. Match the processing guarantee to the domain.

Check: fencing and duplicates​

Quiz

Fencing and processing guarantees

1 / 2

Why should workers verify the caller is still the lock owner (and often the resource version)?

Pick an answer to see if you are right.

Key takeaways
  • Prefer a singleton when simplicity beats continuous availability; use leader election when interactive SLAs need multi-replica ownership.
  • Build locks/leases on consensus KV (etcd/ZooKeeper/Consul) with CAS + TTL + versions — do not hand-roll Raft for app locks.
  • TTL recovers from dead holders; versions stop stale unlocks and fence delayed requests.
  • Renewable leases (renew every ttl/2) give long-lived leaders with fast failover.
  • Assume brief dual-leader windows — workers must verify owner (+ version); choose at-most-once vs at-least-once consciously.
  • Prefer battle-tested lock libraries in your language when available.

Further Reading​