Skip to main content

Monitoring and Observability Patterns

Adapted from: Brendan Burns, Designing Distributed Systems (O'Reilly), Chapter 14 — rewritten and condensed for teaching.

Services run 24/7 across time zones. Reliability means noticing problems before customers do — and having enough signal to understand why. Four building blocks show up in almost every observability stack:

ConceptRole
LoggingOne execution / one place in code
MetricsAggregates over many requests over time
AlertingRules that interrupt you when something crosses a line
TracingRebuild one user request across many services
flowchart LR
  APP[Service]
  LOG[Logs]
  MET[Metrics]
  ALERT[Alerts]
  TRACE[Traces]
  APP --> LOG
  APP --> MET
  APP --> TRACE
  LOG --> ALERT
  MET --> ALERT

Logging​

At heart, logging is still “something happened here with this data” — printf with better manners. Volume is the enemy: useful while debugging a hot component; spam during healthy steady state (and expensive to store/search).

Libraries usually add:

  • Timestamps — order and deltas between lines
  • Request context — thread / user / request IDs so parallel work is separable
  • Levels — filter noise without deleting code paths

Typical levels:

LevelUse
DebugVerbose; on only when investigating
InfoNormal operational breadcrumbs
WarningConcerning but not fatal
ErrorSomething bad; someone should look soon
FatalUnrecoverable; process exits after the line

Default runtime level is often Info or Warning; flip Debug on for a class/component when an incident needs detail (conditional logging). Be careful: turning on heavy debug for every request can cause the next outage via latency.

What to log: errors/exceptions almost always; “weird” cases (a path that took 30s); not a play-by-play of every function. Ask: what would I wish I had logged when debugging the hardest bug in this code?

Check: logging​

Quiz

Logging practices

1 / 2

How do logs differ from metrics?

Pick an answer to see if you are right.

Hands On: Leveled log filter​

Metrics​

Metrics characterize general execution as time series: instantaneous state and how it changes.

Three useful shapes (Prometheus-style vocabulary):

TypeIdeaExamples
HistogramDistribution over an intervalLatency, request/response size
CountMonotonic integralTotal requests served
Value (gauge)Up and downCPU %, memory bytes

Counts often become rates (derivative): requests now minus requests one minute ago → requests/minute. Tiny windows (e.g. per-ms) look spiky; average the rate over a sensible window.

Prometheus is primarily pull/scrape: the server asks apps for the current metric snapshot. Short-lived batch jobs may finish before a scrape — use a push gateway so jobs push metrics that Prometheus still scrapes.

Basic request monitoring​

Start with request count (interesting as a rate), then enrich with labels like HTTP status (200, 404, 500). Labels must be a small discrete set — do not put raw latency or unbounded customer IDs on a counter label.

Latency is its own histogram (p50 vs p99 matter). You can still label latency histograms by status class to compare error vs success latency.

Advanced request monitoring​

  • Request/response size histograms (correlate with slowness; watch growth over time)
  • Split queue wait vs processing latency — scale out vs optimize code
  • Coarse geography as a label; fine customer analytics often via log search (metrics stay cheap aggregates; logs stay rich)

Check: metrics​

Quiz

Metrics shapes

1 / 2

Why is HTTP status a good Prometheus label on a request counter, but raw latency is not?

Pick an answer to see if you are right.

Hands On: Rates, error ratio, latency buckets​

Alerting​

Logs and metrics help after you know something is wrong. Alerting tells you to look — ideally before a customer-reported incident (CRI).

Basic alerting​

Rules on metric queries + static thresholds, e.g.:

  • p90 latency > 500ms
  • HTTP 500s > 1% of requests

Those thresholds effectively encode your SLO: what “good enough” means. Tune for users and on-call health — constant pages and false alarms cause burnout and ignored pages. Alerting is lifelong maintenance as the system and monitors evolve.

Not every metric needs an alert; many exist only for dashboards and debugging. Application quality still tracks alert quality.

Anomaly detection​

Static global thresholds miss 100% outage for 1% of users while the fleet average looks fine. Anomaly / per-cohort detection is harder but increasingly necessary as systems grow.

Check: alerting​

Quiz

Alert design

1 / 2

Why treat alert thresholds as tied to SLOs?

Pick an answer to see if you are right.

Hands On: Should we page?​

Tracing​

Per-service logs and metrics show all traffic mixed together. Tracing reconstructs one user request across microservices.

Practical approach:

  1. Mint a correlation / trace ID at the edge (random large ID, or hash of stable request traits).
  2. Attach it to every log/metric emission.
  3. Propagate it on outbound calls (e.g. HTTP header x-my-apps-correlation-id) into each service’s request context.

Prefer OpenTelemetry SDKs and backends over a one-off home-grown stack — vendor-agnostic instrumentation is already a solved community problem.

flowchart LR
  EDGE[Edge + trace id]
  A[Service A]
  B[Service B]
  C[Service C]
  EDGE -->|header| A --> B --> C

Check: tracing​

Quiz

Distributed tracing

1 / 2

What problem does a correlation ID solve that per-service metrics alone do not?

Pick an answer to see if you are right.

Hands On: Propagate and group by trace​

Aggregating Information​

Observability data grows fast. Aggregation (and tiering) keeps it queryable and affordable.

  • Cross-replica log search — ktail-style fan-in, or ship to Elasticsearch / cloud log search
  • Prefer managed log and metrics services for something this critical
  • Downsample metrics (e.g. 1s → 1m after a week; store averages not one arbitrary sample)
  • For logs: keep errors, sample a fraction, or build app-specific compressions
  • Cold storage when you need long retention / compliance but rare access — slower queries, lower cost; some SaaS products tier automatically

Check: aggregation​

Quiz

Aggregation and retention

1 / 2

Why downsample old high-resolution metrics?

Pick an answer to see if you are right.

Hands On: Downsample a time series​

Key takeaways
  • Observability rests on logs, metrics, alerts, and traces — each answers a different question.
  • Log levels + context; metric histograms / counts / gauges, labels carefully, pull + push gateway for batch.
  • Alerts encode SLOs; watch on-call load and cohort blind spots (anomaly detection).
  • Trace IDs (prefer OpenTelemetry) rebuild one request across services; aggregate and downsample/tier storage so signal stays affordable.

Further Reading​