Distributed Logging
On one machine, "check the logs" means one file. Across dozens of services, it needs a whole pipeline.
Intermediate
TopNotchNote
Notes that matter
why logging is harder once a system is distributed
On a single machine, "check the logs" means opening one file. Once PlateRoute's order flow touches Order, Payment, Dispatch, and Notification services — each potentially running on many instances — a single failed order's story is scattered across dozens of log files on dozens of machines. Distributed logging is the practice of centralizing those logs so a single failure can actually be traced end to end.
structured logging and correlation IDs
A plain text log line ("order failed") is nearly useless once thousands of services are logging simultaneously — there's no way to filter or correlate it with anything else. Two practices fix this:
| Practice | What it enables |
|---|---|
| Structured logging | Log entries as machine-parseable key-value data (JSON) instead of free text, so a logging system can filter, aggregate, and query them precisely. |
| Correlation / trace IDs | A single ID generated when a request first enters the system, passed along to every downstream service call it triggers, and included in every log line along the way — turning a scattered mess of log lines back into one coherent story for that request. |
{"ts":"2026-08-09T14:02:11Z","service":"payment","level":"error",
"trace_id":"7f3a9e21","order_id":482,"msg":"card declined"}
With a trace ID, an engineer investigating order 482's failure can query for
trace_id=7f3a9e21 across every service's logs and see the entire request's path in order, instead of manually cross-referencing timestamps across four separate log files.architecture: getting logs off the box and into one place
| Component | Role |
|---|---|
| Log agent | Runs alongside each service, tailing its log output and forwarding entries centrally — so logs survive even if the machine that generated them is later terminated. |
| Ingestion pipeline | Receives logs from every agent, often buffered through a message queue to absorb bursts without dropping entries. |
| Log storage | A search-optimized store (see Distributed Search) — logs are fundamentally full-text, filterable data, which is exactly what search indexes are built for. |
| Query / dashboard layer | Lets engineers search, filter, and build alerts on log content. |
Logging is I/O-intensive and, done naively, can itself become a performance bottleneck for the service producing the logs — which is why log agents typically write asynchronously and buffer locally rather than blocking the application on every log line.
retention: logs can't be kept forever
Log volume at scale is enormous, and keeping every log line indefinitely isn't affordable or useful. A tiered retention policy is standard: recent logs (say, the last 7-30 days) stay in fast, searchable storage for active debugging; older logs get moved to cheaper cold storage or aggregated into summary metrics and then discarded, keeping only what's needed for compliance or long-term trend analysis.
logging vs. monitoring — different questions
Monitoring answers "is something wrong right now" with aggregated numeric metrics. Logging answers "what exactly happened, in what order" with detailed, per-event records. A monitoring alert ("payment error rate spiked") is what tells an engineer to go looking; the correlated logs are what tell them why. Neither replaces the other.