Skip to main content

Sharded Services

Adapted from: Bilgin Ibryam & Roland Huß, Kubernetes Patterns (O'Reilly, 2019), Chapter 7 — rewritten and condensed for teaching.

Replicated Load-Balanced Services spread identical replicas behind a load balancer — any replica can serve any request. Sharding sends particular requests to particular groups of machines instead. Each shard holds a disjoint slice of data or traffic.

flowchart LR
  REQ[Requests]
  ROUTER[Shard router]
  S0[Shard 0]
  S1[Shard 1]
  S2[Shard 2]
  REQ --> ROUTER
  ROUTER --> S0
  ROUTER --> S1
  ROUTER --> S2

Sharding helps when:

  • Data exceeds one machine — caches, game worlds, databases.
  • Isolation matters — a poison request that crashes replicas may take down one shard, not the entire fleet.

Ambassadors showed how a Pod-local proxy routes to shards; this chapter focuses on building the sharded tier itself and choosing shard keys and hash functions.

Sharded Caching​

A sharded cache sits between users and your app tier.

flowchart LR
  USER[Users]
  CACHE[Sharded cache]
  APP[App tier]
  USER --> CACHE
  CACHE --> APP

Why replicate is not enough for big working sets​

Example: each cache node has 10 GB RAM and handles 100 RPS. Your working set is 200 GB and you need 1,000 RPS.

DesignRPS capacityUnique data cached
10 replicated caches (Ch. 6)10 × 100 = 1,000 ✓~10 GB each → 5% of 200 GB total
10-way sharded cache10 × 100 = 1,000 ✓10 × 10 GB → 50% of 200 GB total

Replicated caches duplicate the same hot keys on every node — great for redundancy, poor for memory utilization. Sharded caches store each key on one shard, dramatically improving hit rate for large catalogs.

Check: sharded vs replicated cache​

Quiz

Why shard a cache?

1 / 1

Ten cache replicas each have 10 GB RAM; the total working set is 200 GB. Why does a replicated (non-sharded) layout only cover about 5% of the data?

Pick an answer to see if you are right.

The Role of the Cache in Performance​

Ask: if the cache fails, what happens to users?

Throughput (hit rate and RPS)​

A serving layer handles 1,000 RPS before returning HTTP 500. Add a cache with 50% hit rate:

  • 2,000 inbound RPS → 1,000 served from cache, 1,000 hit the origin → system stays within capacity.
  • Cache failure doubles origin load → half of users may fail.

If the cache is critical, rate the service below theoretical max (e.g. 1,500 RPS instead of 2,000) so you survive losing half the cache shards.

Latency​

A 25% hit rate cache returning in 10 ms vs 100 ms origin work lowers average latency (e.g. to ~77.5 ms). Cache misses are slower but usually tolerable unless queues back up. Load-test with and without the cache.

Warmup and rollouts​

A new empty shard has 0% hit rate until populated. Rolling a sharded cache temporarily reduces effective capacity — unlike replicated caches where a new replica can share load immediately. Mitigations: replicated shards (each shard is its own small replicated service) or careful rollout timing.

Replicated Sharded Caches​

Combine patterns: each shard is implemented as a replicated service (Ch. 6), not a single server.

  • Shard failure does not remove an entire partition from the fleet.
  • Rollouts during peak traffic become safer with over-provisioned shard capacity.
  • Hot shards can scale independently (hot sharding) — replicate a busy shard onto more machines while cooler shards share hardware.

Trade-off: more moving parts than simple sharding.

Check: cache criticality and shards​

Quiz

Sharded cache behavior

1 / 1

Why does losing one shard in a sharded cache hurt more than losing one replica in a fully replicated cache?

Pick an answer to see if you are right.

Hands On: Sharded Memcache with an Ambassador​

Same shape as sharded Redis in Ambassadors: StatefulSet shards, headless Service DNS, twemproxy routing, ConfigMap config.

Memcache StatefulSet​

Complete the StatefulSet for three memcached shards. serviceName must match the headless Service you create next; Pods need the app: memcache label.

Headless Service​

Pods should resolve as memcache-0.memcache, memcache-1.memcache, etc.

twemproxy config (Pod ambassador)​

Route memcache protocol on loopback so the app container in the same Pod talks to 127.0.0.1:11211. List all three shard DNS names under servers.

Shared Shard Router (Alternative to Ambassador)​

Instead of a twemproxy ambassador per app Pod, deploy a replicated shard router as its own tier:

  • Pros: simpler client — one DNS name; no sidecar in every Pod.
  • Cons: shared bottleneck to scale; extra network hop and bandwidth.

The twemproxy config difference: listen on 0.0.0.0:11211 (all interfaces) instead of loopback, then front it with a Service load balancer.

Complete the Deployment (three shared-twemproxy replicas, mount shared-twem-config) and Service (shard-router-service, selector app: shared-twemproxy, port 11211) using the same patterns as Replicated Load-Balanced Services.

Sharding Functions​

Pick shard for request Req:

Shard = ShardingFunction(Req)

Common approach: hash(key) % num_shards

Requirements:

  • Determinism — same input → same shard.
  • Uniformity — spread load evenly across shards.

Selecting the shard key​

Do not hash the entire request object if most fields do not affect the response.

Key choiceEffect
hash(full request)Different client IP / time → different shards for identical cacheable responses — too general
hash(request.path)Good when path alone defines the response
hash(ip, path)French vs US IPs need different content, but two French IPs land on different shards — too specific
hash(country(ip), path)Groups geographically equivalent users while splitting countries

The right key requires understanding which request attributes actually change the response. See also Partitioning of Key-Value Data for database partition keys.

Check: shard keys​

Quiz

Choosing a shard key

1 / 2

Two HTTP GETs differ only by client IP and timestamp; the response depends only on the URL path. Which shard key is best for a cache in front of this API?

Pick an answer to see if you are right.

Consistent Hashing​

Changing from hash(Req) % 10 to hash(Req) % 11 remaps most keys — in a cache, that looks like a mass miss event (similar to total cache failure).

Consistent hashing limits remapping when shard count changes — moving from 10 to 11 shards remaps roughly K/11 keys instead of nearly all of them. twemproxy's ketama distribution uses this idea.

Hands On: nginx consistent hash HTTP proxy​

Shard HTTP requests by full URI ($request_uri — path + query; add cookies or locale if responses vary by them). Use nginx hash $request_uri consistent in the upstream block.

Beyond Caching: Hot Shards and Isolation​

Sharded replicated serving applies to any dataset larger than one machine — open-world game servers shard by player location; databases shard by user id or range.

Hot shards happen when traffic is uneven (a viral photo, a celebrity account). With replicated shards, scale the hot partition independently — even colocate cool shards and replicate the hot one onto extra machines (autoscaling per shard).

Poison or crash-inducing requests are isolated to one shard — the blast radius is a fraction of total traffic, not the whole fleet.

flowchart TB
  subgraph before["Even load"]
    B0[Shard A]
    B1[Shard B]
    B2[Shard C]
  end
  subgraph after["Hot shard A"]
    A1[Shard A replica 1]
    A2[Shard A replica 2]
    BC[Shard B + C colocated]
  end
Key takeaways
  • Sharding routes requests to partition-specific server groups — unlike replication, each shard holds different data.
  • Sharded caches maximize memory for large working sets; replicated caches maximize duplicate hot data for RPS and redundancy.
  • Hit rate ties cache health to origin capacity and user latency — design headroom for shard loss and warmup after deploys.
  • Replicated shards combine both patterns: each partition is its own small HA replica set.
  • Shard keys must reflect what actually changes the response — not every request field belongs in the hash.
  • Consistent hashing (ketama, nginx consistent) limits key churn when shard counts change.
  • Deploy shard routers as Pod ambassadors (localhost) or shared Services (0.0.0.0) — simpler ops vs extra hop.

Further Reading​