Skip to main content

Replicated Load-Balanced Services

Adapted from: Bilgin Ibryam & Roland Huß, Kubernetes Patterns (O'Reilly, 2019), Chapter 6 — rewritten and condensed for teaching.

The simplest distributed pattern — and one most teams already know — is a replicated load-balanced service: several identical servers behind a load balancer, any replica able to serve any client request.

flowchart LR
  CLIENT[Clients]
  LB[Load balancer]
  R1[Replica 1]
  R2[Replica 2]
  R3[Replica 3]
  CLIENT --> LB
  LB --> R1
  LB --> R2
  LB --> R3

This chapter walks through deploying that pattern in Kubernetes, then stacking additional replicated tiers for caching and TLS termination.

Stateless Services​

A stateless service does not depend on saved per-request state on a particular instance. Any replica can handle any request — static file servers, aggregators that fan out to backends, dictionary lookup APIs, and many HTTP APIs fit this model.

Replicas serve two goals:

  1. Redundancy — survive rollouts and crashes without user-visible downtime.
  2. Scale — add replicas as traffic grows (horizontal scaling).

Why you need at least two replicas for a real SLA​

A single instance makes aggressive uptime targets nearly impossible during deploys.

For 99.9% availability you get about 1.4 minutes of downtime per day (24 × 60 × 0.001). If you deploy once per day, each rollout must finish in under 1.4 minutes — and that assumes the process never crashes from bugs. Deploy hourly and each rollout must complete in about 3.6 seconds to stay within budget on a single node.

Two replicas behind a load balancer let you roll one instance while the other keeps serving — users never notice a healthy rollout or a single crash.

Check: stateless replication​

Quiz

Stateless replicated services

1 / 2

Why is a single replica a poor fit for a 99.9% availability SLA when you deploy frequently?

Pick an answer to see if you are right.

Readiness Probes for Load Balancing​

Replicating and load-balancing is not enough. The orchestrator and load balancer need to know when a replica is ready for traffic — not just alive.

ProbeQuestionTypical action if it fails
LivenessIs the process alive?Restart the container
ReadinessCan this instance serve requests right now?Remove it from the load balancer pool

Many apps start quickly but are not ready immediately: downloading data, connecting to databases, loading plugins. The dictionary server in the hands-on section starts serving HTTP immediately but only reports ready after an ~8 MB dictionary finishes downloading.

Build a readiness endpoint (e.g. GET /ready) and wire it into the Pod spec so traffic only hits warmed replicas. See also Important Distributed System Concepts for the liveness/readiness distinction.

Check: readiness probes​

Quiz

Readiness vs liveness

1 / 1

A container is downloading a large file it needs before it can answer requests correctly. It is alive but not yet usable. Which probe should fail?

Pick an answer to see if you are right.

Hands On: Dictionary Server Deployment​

We'll deploy brendanburns/dictionary-server — visit /dog for a definition; /ready reports when the dictionary has loaded.

Local smoke test (optional):

docker run -p 8080:8080 brendanburns/dictionary-server
# http://localhost:8080/dog

Deployment with readiness​

Complete the Deployment: run three replicas, label Pods so a Service can select them, and add an HTTP readiness probe on /ready (port 8080, sensible initialDelaySeconds / periodSeconds).

Service load balancer​

Add a Service that selects the dictionary Pods and exposes port 8080 (with targetPort matching the container).

The Service gives clients a stable DNS name independent of any single Pod IP.

Session-Tracked Services​

Round-robin spreads load evenly but sends each user's requests to random replicas. Sometimes you need session stickiness — all requests from one user hit the same replica (in-memory cache, long-lived interaction state).

IP hashing (source + destination IP) keeps a client on one backend while IPs stay constant. Inside a cluster this works with internal IPs; NAT breaks it for external clients — use cookies or application headers instead.

When replica count changes, remapping every client is expensive. Consistent hashing minimizes how many sessions move when you scale up or down.

Check: session tracking​

Quiz

Session-tracked services

1 / 1

Why is IP-based session affinity often a poor choice for external users behind NAT?

Pick an answer to see if you are right.

Application-Layer Replicated Services: A Caching Tier​

Even stateless app code can be expensive (database queries, rendering). An HTTP cache (e.g. Varnish) sits between clients and your app tier: identical requests are answered from memory after the first miss.

flowchart LR
  CLIENT[Clients]
  CACHE[Varnish tier]
  APP[Dictionary tier]
  CLIENT --> CACHE
  CACHE --> APP

Sidecar cache vs dedicated cache tier​

You could run Varnish as a sidecar beside each app Pod — simple, but every page is cached on every replica. That duplicates memory and lowers hit rate.

Usually you want few large cache replicas and many smaller app replicas (Node.js often uses one core per process). Deploy caching as a second replicated tier above the app tier, each with its own Service.

Caching breaks naive IP stickiness

With a cache tier, backends see the cache's IP, not the end user. IP affinity at the app layer may send all traffic to one app Pod. Use cookies or headers for session tracking behind a cache.

Hands On: Varnish Caching Layer​

The dictionary Service is reachable in-cluster as dictionary-server-service:8080. Complete the Varnish config, Deployment, and Service.

Varnish backend​

Point the default backend at the dictionary Service on port 8080.

Create the ConfigMap: kubectl create configmap varnish-config --from-file=default.vcl

Varnish Deployment​

Deploy two cache replicas with a 2Gi memory request, mount varnish-config at /etc/varnish-config, and run varnishd with -f /etc/varnish-config/default.vcl listening on 0.0.0.0:8080.

Varnish Service​

Expose the cache tier: select app: varnish-cache, listen on port 80, forward to container port 8080.

Expanding the Edge: Rate Limiting and TLS​

HTTP reverse proxies can also provide rate limiting (Varnish throttle modules, HTTP 429 responses, custom X-RateLimit-Remaining headers) and are a natural place for anonymous vs authenticated quotas. For mission-critical public APIs, add cloud DDoS protection beyond what a proxy alone provides.

Varnish does not terminate TLS. A third replicated tier of nginx servers often handles HTTPS at the edge, using per-layer certificates (edge cert ≠ internal service certs) so each tier rolls independently.

Complete the nginx SSL server block: terminate TLS on 443 and proxy_pass to the Varnish Service on port 80, forwarding standard headers (Host, X-Forwarded-For, X-Forwarded-Proto, X-Real-IP).

Upload certs as a Secret (kubectl create secret tls ssl --cert=server.crt --key=server.key), mount the nginx ConfigMap and Secret in a fourth Deployment, and expose it with a type: LoadBalancer Service on port 443 — same shape as the tiers above.

flowchart TB
  CLIENT[Internet clients]
  NGX[nginx SSL tier]
  VAR[Varnish cache tier]
  DICT[Dictionary app tier]
  CLIENT -->|HTTPS| NGX
  NGX -->|HTTP| VAR
  VAR --> DICT
Key takeaways
  • Replicated load-balanced services put identical replicas behind a load balancer for redundancy and horizontal scale.
  • Two or more replicas make realistic SLAs achievable during rollouts; one replica forces impossibly fast deploys.
  • Readiness probes gate traffic until an instance is initialized; liveness probes restart broken processes — do not conflate them.
  • Session stickiness via IP hash works in-cluster; prefer cookies/headers for external users and behind caches.
  • Dedicated cache tiers (few large replicas) beat per-Pod cache sidecars for hit rate; watch stickiness when traffic comes from the cache layer's IP.
  • A full edge stack often stacks nginx (TLS) → Varnish (cache/rate limit) → app tier, each as its own replicated Service.

Further Reading​