Skip to main content

Scalable Systems

Every system that outgrows a single machine — or a single, simple data model — runs into the same handful of hard problems: how to represent data so it stays fast to query as it grows, how to keep multiple copies of that data consistent, how to split it across machines without losing correctness, and how to reason clearly about what "correct" even means once concurrency and network failures are in the picture. This section builds up that body of knowledge from first principles, grounded in how real, large-scale data systems are actually built.

Designing Data-Intensive Applications​

Foundational knowledge for anyone building systems that must reliably store, query, and process data at scale — data models, storage engines, replication, partitioning, transactions, and the consistency guarantees that hold it all together.

  • Graph-Like Data Models — the property graph model, the Cypher query language, expressing graph traversals in SQL via recursive CTEs, the triple-store model and SPARQL, Datalog, and how graph databases differ from the old CODASYL network model
  • Hash Indexes and LSM-Trees — the append-only log at the heart of every storage engine, hash indexes and Bitcask, SSTables and the sparse in-memory index, memtables and write-ahead logs, and LSM-trees (LevelDB/RocksDB, Bloom filters, compaction strategies)
  • B-Trees and Other Indexing Structures — B-tree pages and branching factor, write-ahead logs and latches for reliability, B-trees vs. LSM-trees (write amplification), secondary indexes, heap files, clustered/covering indexes, multi-dimensional indexes (R-trees), full-text/fuzzy search, and in-memory databases
  • Formats for Encoding Data — backward/forward compatibility, why language-specific encodings are risky, JSON/XML pitfalls, Thrift/Protocol Buffers (field tags and schema evolution rules), and Avro (writer's/reader's schema resolution)
  • Modes of Dataflow — dataflow through databases (data outlives code), REST vs. SOAP, the problems with RPC (with an idempotent-retry code exercise), and message-passing dataflow (brokers and distributed actor frameworks)

Distributed Data​

How to keep the same data correct and available across many machines — replication, partitioning, transactions, and the consistency and consensus guarantees that make distributed systems reason-about-able at all.

  • Leaders and Followers — leader-based replication, synchronous vs. asynchronous replication, setting up new followers, handling node outages and failover, and how replication logs are actually implemented (statement-based, WAL shipping, logical/row-based, trigger-based)
  • Problems with Replication Lag — eventual consistency, read-after-write consistency, monotonic reads (with a sticky-routing code exercise), and consistent prefix reads
  • Multi-Leader Replication — when multiple leaders make sense (multi-datacenter, offline clients, collaborative editing), handling write conflicts (with a cart-merging code exercise), automatic conflict resolution (CRDTs), and replication topologies
  • Leaderless Replication — Dynamo-style writes/reads, read repair and anti-entropy, quorums (n/w/r) and their real limitations, sloppy quorums and hinted handoff, and detecting concurrent writes (happens-before, last write wins, version vectors, with a sibling-resolution code exercise)
  • Partitioning of Key-Value Data — why partitioning is necessary, key range partitioning, hash partitioning (with a partition-assignment code exercise), and relieving hot spots caused by a single overloaded key
  • Partitioning and Secondary Indexes — document-partitioned (local) indexes and scatter/gather (with a code exercise), and term-partitioned (global) indexes and their async-update trade-off
  • Rebalancing and Request Routing — why hash-mod-N rebalancing fails (with a code exercise quantifying the churn), fixed/dynamic/proportional partitioning strategies, automatic vs. manual rebalancing, and request routing (ZooKeeper vs. gossip)
  • Faults, Partial Failures, and Unreliable Networks — why partial failure is the defining trait of distributed systems, HPC vs. cloud fault philosophies, the six ways a network request can silently fail, detecting faults, timeouts and queueing (with a fault-detector code exercise), and circuit- vs. packet-switched networks
  • Unreliable Clocks — time-of-day vs. monotonic clocks, why NTP synchronization is less reliable than you'd hope, the dangers of last-write-wins timestamps, TrueTime-style confidence intervals (with a code exercise), and process pauses (GC, VM suspension, hard real-time systems)
  • Knowledge, Truth, and Lies — why truth is defined by quorum majority, fencing tokens for safe leader/lock handoff (with a code exercise), Byzantine faults and when they actually matter, and system models (safety vs. liveness properties)
  • Linearizability — the recency guarantee precisely defined, linearizability vs. serializability, where it's actually needed (locking, uniqueness constraints, cross-channel races), which replication methods can deliver it, and the CAP theorem properly understood
  • Ordering Guarantees — causality as a partial order vs. linearizability's total order, Lamport timestamps (with a code exercise), why timestamp ordering alone isn't sufficient, and total order broadcast (equivalent to both linearizable storage and consensus itself)
  • Distributed Transactions and Two-Phase Commit — the FLP impossibility result (and why it doesn't doom real systems), 2PC's system of promises (with a vote-tally code exercise), coordinator failure and in-doubt transactions, XA transactions, and the real operational costs of distributed transactions
  • Fault-Tolerant Consensus — the four formal consensus properties, why consensus algorithms are usually total order broadcast algorithms, epoch numbers and overlapping quorums (with a code exercise) resolving the leader-election chicken-and-egg problem, and ZooKeeper/etcd as coordination services

Derived Data​

How large, derived datasets — search indexes, recommendation tables, caches, analytics stores — actually get built and kept in sync with a system of record: batch processing, messaging systems and partitioned logs, change data capture, event sourcing, and the correctness and ethical questions that come with building on top of all of it.

  • Batch Processing with Unix and MapReduce — services vs. batch vs. stream processing, the Unix philosophy (uniform interfaces, separation of logic/wiring, transparency), and MapReduce/HDFS fundamentals (the shuffle, chained workflows) with a top-N-URL-counter code exercise
  • Joins in MapReduce — reduce-side sort-merge joins and GROUP BY, handling skewed/hot keys, and map-side joins (broadcast, partitioned, merge) with two code exercises
  • Beyond MapReduce: Dataflow Engines and Graphs — batch job outputs (search indexes, bulk-loaded key-value stores, human fault tolerance), Hadoop vs. MPP databases, dataflow engines (Spark/Tez/Flink) and recomputation-based fault tolerance, and the Pregel/BSP graph model (with a superstep code exercise)
  • Transmitting Event Streams — direct messaging vs. message brokers, load balancing vs. fan-out, acknowledgments and the message-reordering problem (with a consumer-offset code exercise), partitioned logs vs. traditional messaging, disk space and slow consumers, and replaying old messages
  • Databases and Streams — the dual-writes race condition, change data capture (implementation strategies, initial snapshots, log compaction, with a compaction code exercise), and event sourcing (deriving current state, commands vs. events, with a replay code exercise)
  • Data Integration and the Limits of Total Order — combining specialized tools by deriving data, derived data vs. distributed transactions, why total ordering stops scaling, and capturing causality across systems (with a causal-reference code exercise)
  • Unifying Batch and Stream Processing — maintaining derived state asynchronously, reprocessing data for application evolution, the lambda architecture and its practical problems, and unifying batch/stream engines (with a lambda-merge code exercise)
  • Unbundling Databases — composing storage technologies, CREATE INDEX as reprocessing (with a code exercise), the meta-database of everything, federated vs. unbundled databases, and what's still missing
  • Designing Applications Around Dataflow — application code as a derivation function, separating app code from state, dataflow vs. ordinary job queues, and stream processors vs. services (with a stream-table-join code exercise)
  • Observing Derived State — the write path vs. read path (with a cache-vs-index code exercise), stateful offline-capable clients, pushing state changes to clients, end-to-end event streams, reads as events, and multi-partition query processing
  • Correctness and Integrity in Dataflow Systems — exactly-once semantics and the end-to-end argument (with a request-deduplication code exercise), enforcing uniqueness constraints via partitioned logs (with a username-claiming code exercise) and multi-partition requests, timeliness vs. integrity, coordination-avoiding design, and auditing/trust-but-verify
  • Ethics of Data-Intensive Applications — bias and accountability in predictive analytics, self-reinforcing feedback loops, surveillance and consent, privacy as a decision right, data as an asset, and lessons from the Industrial Revolution

Distributed Systems​

The foundational vocabulary and trade-offs distributed systems keep coming back to — APIs and RPCs, latency and reliability measurement, percentiles, idempotency, delivery semantics, relational and data consistency, the CAP theorem, and how orchestrators like Kubernetes use health checks to manage running services.

  • Important Distributed System Concepts — APIs and RPCs (sync vs. async, IDLs, SLOs), latency and attribution, defining reliability (the HTTP status-code gray area), why percentiles beat averages, idempotency, at-least-once vs. at-most-once delivery, relational integrity, strong vs. eventual consistency and the CAP theorem, and orchestration (liveness vs. readiness health checks)
  • The Sidecar Pattern — co-scheduled containers in a Pod, TLS termination for legacy HTTP apps, dynamic configuration via shared volumes, modular utility sidecars (PID namespace), git-sync PaaS workflows, and designing reusable sidecar APIs
  • Ambassadors — client-side proxies that broker sharded storage, environment-specific service discovery, and A/B request splitting; hands-on StatefulSet + twemproxy and nginx experiment manifests
  • Replicated Load-Balanced Services — stateless replicas behind a Service, readiness probes, session stickiness, Varnish caching tier, and nginx SSL termination
  • Sharded Services — sharded caches vs replicated caches, memcache + twemproxy, shared shard routers, shard keys, consistent hashing, and hot shards
  • Scatter/Gather — parallelize one request across leaves, term vs leaf sharding for search, stragglers and leaf count, replicated shards for reliability
  • Functions and Event-Driven Processing — FaaS vs serverless, when functions fit, decorators, async event handlers, and event pipelines
  • Ownership Election — singletons vs leaders, CAS/TTL locks on etcd-style stores, renewable leases, and fencing against dual leaders
  • Work Queue Systems — generic source/worker containers, Kubernetes Jobs, dynamic scaling math, and multiworker composition
  • Event-Driven Batch Processing — workflow DAGs: copier/filter/splitter/sharder/merger, pub/sub, work stealing, and poison retries
  • Coordinated Batch Processing — join barriers, reduce (count/sum/histogram), and composing them in an image pipeline
  • Monitoring and Observability Patterns — logs, metrics, alerts, traces, Prometheus pull/push, and aggregation/downsampling
  • Common Failure Patterns — thundering herd, silent zeros, client-error blind spots, versioning, optional caches, runaway deletes, obsolete work, second-system rewrites

System Design Interview​

Interview-oriented walkthroughs that grow a design step by step — naming each component, the bottleneck it removes, and the trade-offs an interviewer expects you to discuss.

  • Scale From Zero to Millions of Users — single-server request flow, SQL vs. NoSQL, vertical vs. horizontal scaling, load balancers, primary/replica replication, the cache tier (with a cache-aside code exercise), CDNs, stateless web tiers (with a sticky-routing code exercise), multi-data-center setups, message queues, logging/metrics/automation, and sharding (with a shard-routing code exercise)
  • Back-of-the-Envelope Estimation — powers of two, latency numbers every programmer should know, availability nines and SLA downtime budgets (with a downtime code exercise), series vs. parallel availability, and estimating QPS, peak QPS, and storage for a Twitter-like service (with an estimator code exercise)
  • A Framework for System Design Interviews — what interviewers look for (and red flags like over-engineering), the four-step process (scope → high-level design → deep dive → wrap-up) worked on a news feed system, dos and don'ts, and 45-minute time allocation (with a time-budget code exercise)
  • Design a Rate Limiter — client vs. server vs. API-gateway placement, the five classic algorithms (token bucket, leaking bucket, fixed window, sliding window log, sliding window counter — with three code exercises), Redis INCR/EXPIRE architecture, Lyft-style rules, HTTP 429 and rate-limit headers, race conditions and synchronization across limiters, edge deployment, and monitoring
  • Design Consistent Hashing — the rehashing problem with hash % N (with a churn-measuring code exercise), the hash ring and clockwise lookup (with a ring-lookup code exercise), adding/removing servers, uneven partitions and skew, virtual nodes (with a full vnode-ring code exercise), finding affected key ranges, and real-world users (Dynamo, Cassandra, Discord, Akamai, Maglev)
  • Design a Key-Value Store — single-server vs. distributed stores, CAP with concrete partition scenarios, partitioning and replica preference lists (with a code exercise), N/W/R quorums (with a code exercise), consistency models, vector clocks for conflict detection (with a code exercise), gossip failure detection, sloppy quorum and hinted handoff, Merkle-tree anti-entropy (with a code exercise), and Cassandra-style write/read paths with Bloom filters
  • Design a Unique ID Generator in Distributed Systems — why AUTO_INCREMENT doesn't scale out, multi-master step-by-k IDs (with a code exercise), UUIDs, Flickr-style ticket servers, Twitter Snowflake's 64-bit layout (with a bit-packing code exercise), timestamp lifetime and custom epochs, sequence overflow and backwards clocks (with a generator code exercise), and section-length tuning
  • Design a URL Shortener — QPS and storage estimates (and a classic double-counting slip), REST endpoints, 301 vs. 302 redirects, sizing a 7-character base-62 code, hash + collision resolution (with a code exercise), base-62 conversion (with a code exercise), comparing the two, and the full shorten/redirect flows with cache and database (with a service code exercise)
  • Design a Web Crawler — crawler use cases and qualities, capacity estimates, the full pipeline (seed URLs, frontier, downloader, DNS, parser, Content Seen?, link extraction with a code exercise, URL filter, URL Seen?), BFS vs. DFS, the URL frontier's politeness back queues (with a code exercise), priority front queues and freshness, robots.txt (with a code exercise), performance and robustness, extensibility, spider traps, and a de-duplicating BFS crawl (with a code exercise)
  • Design a Notification System — APNs/FCM push, SMS and email providers, contact-info gathering, evolving from a single server to per-channel queues and workers, reliability (notification log, retries, why exactly-once is impossible), templates (with a code exercise), opt-in settings and per-user rate limiting (with a dispatcher code exercise), retrying workers (with a code exercise), security, queue monitoring, and event tracking
  • Design a News Feed System — publish and retrieve APIs, feed publishing and building flows, fanout on write vs. read and the celebrity hotkey problem (with a cost code exercise), the hybrid model and the fanout pipeline with mutes and capped ID-only caches (with a hybrid-feed code exercise), feed hydration (with a code exercise), and the five-layer cache architecture
  • Design a Chat System — polling vs. long polling vs. WebSocket, stateless/stateful/third-party components, KV-store chat history and message IDs, service discovery, 1:1 and group message flows (with a group-inbox code exercise), multi-device sync (with a code exercise), and heartbeat presence with pub/sub fanout (with a code exercise)
  • Design a Search Autocomplete System — typeahead QPS estimates, tries with top-k cached per node (with a code exercise that handles decreasing frequencies), the offline data-gathering pipeline (with a weekly-aggregation code exercise), Trie Cache/DB, browser caching and sampling, filter-layer deletes, and traffic-aware prefix sharding (with a shard-map code exercise)
  • Design YouTube — storage and CDN cost estimates, upload and streaming flows, containers vs. codecs, DAG-based transcoding (with a stage-scheduling code exercise), the resource manager (with a scheduler code exercise), parallel/resumable uploads, pre-signed URLs (with an HMAC code exercise), DRM, long-tail cost savings, and an error-handling playbook
  • Design Google Drive — capacity estimates, APIs with resumable upload, evolving from one server to S3 with cross-region replication, first-processed-wins sync conflicts (with an optimistic-concurrency code exercise), block servers and delta sync (with a code exercise), strong consistency with a relational metadata DB, the metadata schema, upload/download flows, long-polling notifications, block dedupe (with a code exercise) and version/cold-storage savings, and failure handling
  • The Learning Continues — how to read an architecture write-up, real-world system readings from Facebook, Amazon, Netflix, Google, Twitter, and others (short links resolved, each mapped to the lesson it extends), and company engineering blogs

Reliable AI Systems​

Building LLM-powered applications that hold up in production: choosing and configuring models, prompting techniques that reduce hallucinations, and grounding answers in authoritative data.

  • Generating Trustworthy Responses with Prompt Engineering — model choice as a reliability decision (reasoning vs. non-reasoning, effort settings), temperature (with a softmax code exercise) and top-p (with a nucleus-filter code exercise), output limits and stop sequences, frequency/presence penalties, seeds and voting, prompt components, zero-shot/few-shot/chain-of-thought/Auto-CoT prompting, self-consistency (with a voting code exercise), tree-of-thought, and a function-calling weather assistant for OpenAI and Claude (with a tool-dispatch validation code exercise)
  • Grounding Outputs with RAG — the freshness problem, retriever/generator architecture, the load→split→embed→store indexing pipeline, a FAISS-backed query pipeline, thresholded retrieval (with a code exercise), LLM judges and RAGAS, why word-overlap grounding scores mislead (with a code exercise), lost-in-the-middle edge ordering (with a code exercise), Haystack metadata filtering and the over-filtering trap (with a code exercise), GraphRAG, agentic RAG, and an end-to-end LangChain pipeline
  • Embeddings and Vector Search — embeddings and cosine similarity, choosing commercial/open-source/domain models (and E5 prefixes), Airbnb's two-tower case study, BM25 from scratch (with a code exercise), hybrid retrieval with reciprocal rank fusion (with a code exercise) and cross-encoder reranking, metadata filtering, content-aware chunking (with a sliding-window code exercise), HNSW/FAISS and vector databases, a Pinecone health-care policy assistant, embedding drift, and int8 quantization (with a code exercise)
  • Fine-Tuning LLMs for Improved Performance — prompting vs. RAG vs. fine-tuning decision framework (with a code exercise), real-world cases (Med-PaLM 2, BloombergGPT, Codex — facts corrected), data collection and chat-format datasets (with a validator code exercise), leakage-proof hash splits (with a code exercise), OpenAI fine-tuning and checkpoints, full fine-tuning vs. LoRA vs. QLoRA with parameter and memory sizing (with a code exercise), a QLoRA customer-support assistant with TRL, early stopping (with a code exercise), and knowledge distillation
  • Creating Effective AI Agents — why agents (static knowledge, no actions, no workflows), short-term memory and context trimming (with a code exercise), persistent SQLite memory with user deletion (with a code exercise), grounded tools with validation and fallback (and the Kelvin bug), cross-referencing with tolerances (with a code exercise), rule-based vs. AI-driven decisions, a ReAct loop with step limits and loop detection (with a code exercise), an agentic RAG travel assistant in Python, guardrails, transparency, production case studies, and framework choices
  • Tool Integration and MCP — the N×M integration problem and MCP's N+M fix, hosts/clients/servers and JSON-RPC over stdio or Streamable HTTP, a FastMCP CSV product-search server (with a word-matching search code exercise that fixes the substring bug), tool discovery via tools/list, hosted MCP with OpenAI and Claude (public URLs required), a JSON-RPC handler (with a code exercise), tool design and chaining, structured errors vs. empty results, retries with exponential backoff (with a code exercise), agent harnesses and sandboxes, and MCP security
  • Multi-Agent Systems — the monolithic agent problem, LangGraph state/nodes/conditional edges (with a mini graph-engine code exercise), structured intent analysis, ShopBot's RAG policy agent, search and vision agents with the source's orchestration bugs fixed, graceful degradation (with a code exercise), per-request state isolation (with a code exercise), deterministic workflow testing with fakes, and LangGraph vs. CrewAI
  • Evaluation and Performance for LLMs and Agents — the four-step hallucination measurement method with GDR/HSS/topic reports (with a code exercise), FActScore (with a code exercise), ROUGE and why it misleads (with a code exercise), LLM-as-a-judge and its biases, red teaming, Phoenix monitoring, streaming vs. full responses, system-level batching vs. the Batch API (source's async bugs fixed), safe semantic caching with thresholds and versioning (with a code exercise), multi-model routing, and agent evaluation of answers, tool calls, and trajectories (with a code exercise)
  • Deploying and Monitoring LLM Applications — LLMOps layers, hosted vs. self-hosted vs. hybrid deployment with break-even economics done properly, a privacy-preserving hybrid router that never leaks sensitive queries on failure (with a code exercise), chat templates and vLLM/TGI for self-hosting, structured logs and tiered alerts, cost spikes vs. efficiency regressions (with a code exercise), explicit and implicit feedback with weekly triage, numeric prompt versioning with pins (with a code exercise), golden-test deploy gates (with a code exercise), shadow testing and canaries (source's blocking async fixed), when to switch models, and Langfuse observability
  • Bias, Privacy, and Responsible AI — the EU AI Act, Colorado's AI Act, and documented failures (Amazon, COMPAS, Obermeyer), four failure modes and layered defenses, auditing logs without name-list inference, the UW résumé-embedding study, whole-word counterfactual swaps (with a code exercise), fairness metrics and the impossibility result (with a code exercise), a fairness loss that actually works and Constitutional AI described accurately, fail-closed safety classification (with a code exercise), LLM privacy leak vectors, Luhn-validated PII redaction (with a code exercise), HIPAA Safe Harbor and GDPR erasure, a corrected SafeMedAssist pipeline, and the six principles of reliable AI

Practice: every lesson's source references come with quizzes. Pass one to unlock that citation, then write short essays with the lessons' keywords and citations in the Essay Lab.

Background reading (references from the source book's introduction; quizzes are in the Essay Lab under "Why Reliability Matters"):

  1. Fortune (2025). MIT report: 95% of generative AI pilots at companies are failing.
  2. Rein, D., et al. (2023). GPQA Diamond (Epoch AI benchmark page).
  3. Legora Newsroom (2026). Legora extends Series D with additional $50 million, welcomes Atlassian and NVentures as investors.
  4. Anthropic (2026). Anthropic raises $30 billion in Series G funding at $380 billion post-money valuation.
  5. The Economic Times (2026). Codex user base grows to 4 million; OpenAI says 1 million added in just 2 weeks.
  6. Panto (2026). Cursor AI Statistics 2026: Users, Revenue, Adoption & Key Growth Metrics.
  7. CodeRabbit (2026). 2025 was the year of AI speed. 2026 will be the year of AI quality.
  8. Faros (2026). Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash.

Planned Topics​

Data Models and Storage​

  • Relational vs. document vs. graph data models — trade-offs and when each fits
  • Column-oriented storage for analytics

Pages coming soon — check back or contribute a page using the template.