Monitoring & Observability for Distributed Systems
Every fault-tolerance technique in this track assumes something is watching for failure. This is that watcher.
Intermediate
TopNotchNote
Notes that matter
finding out something is wrong before a customer tells you
Every fault-tolerance technique covered so far in this track — replication, checkpointing — assumes something is watching for failure in the first place. Monitoring is that watcher: continuously collecting, interpreting, and displaying data about a system's health, so a problem can be caught and fixed before it snowballs, ideally before a user ever notices.
Two broad postures: reactive monitoring reacts after a failure has already caused visible downtime; proactive monitoring watches leading indicators (rising latency, shrinking free disk) to catch trouble before it becomes an outage. Mature systems lean heavily proactive.
picking metrics and thresholds
A metric is only useful if it's specific and has a defined unit — "the search service feels slow" isn't a metric; "p99 search latency in milliseconds" is. Once a metric is defined, it needs a threshold that separates "normal" from "needs attention," and an owner who gets alerted when it crosses that threshold.
| Metric category | Examples |
|---|---|
| Resource usage | CPU load, memory usage, disk I/O latency, free disk space |
| Service-level | Requests per second, error rate, p50/p95/p99 latency |
| Business-level | Orders placed per minute, payment success rate |
push vs. pull metric collection
| Model | How it works | Trade-off |
|---|---|---|
| Pull | The monitoring system periodically scrapes each service's metrics endpoint. | The monitoring system controls collection frequency and knows immediately if a service stops responding at all; requires every service to expose a scrape-able endpoint. |
| Push | Each service proactively sends its metrics to the monitoring system on its own schedule. | Works well for short-lived jobs that might not exist long enough to be scraped; adds steady background load on the monitoring system's ingest path from every service pushing independently. |
architecture of a monitoring system
| Component | Role |
|---|---|
| Data collector | Fetches or receives metrics from every monitored service and writes them into storage. |
| Time-series storage | A database purpose-built for append-heavy, time-ordered numeric data — see Specialized Data Stores. |
| Querying / dashboard service | Lets engineers explore metrics visually and build dashboards. |
| Alert manager | Evaluates rules against incoming metrics and notifies the right people when thresholds are violated. |
Because the monitoring system itself can become a single point of failure (ironically, an outage nobody gets alerted to because the alerting system is what's down), production deployments typically run a secondary monitoring tier — a lightweight pull-based tier collecting from primary services, feeding into a push-based path to a more durable central system — so a failure in one tier doesn't blind the whole pipeline.
client-side monitoring: seeing what the server can't
Server-side monitoring has zero visibility into failures that happen entirely on the client's path — a broken DNS resolution, a routing failure between the user and the service, or a misbehaving CDN edge node. For PlateRoute, a customer whose ISP is dropping packets to the origin datacenter never generates a server-side error at all — from the server's point of view, nothing happened.
The fix is an agent embedded in the client app that actively performs its own reachability checks and reports failures to an independent collector service. The collector is kept separate from the primary service specifically so a client that can't reach the primary service can still successfully report that fact — reporting to the very service that's failing would defeat the purpose.
what monitoring feeds into
Monitoring data is the raw material for two things covered elsewhere in this track: distributed logging for detailed post-incident debugging, and the on-call alerting that turns a metric crossing a threshold into a page. Neither is useful without the other — logs without metrics mean nobody knows when to go looking; metrics without logs mean knowing something's wrong without knowing why.