Observability: logs, metrics and traces

Your code works in tests. Then, in production, some users see errors, but only some, only sometimes, and never when you look. You can’t attach a debugger to a live service handling thousands of requests across a dozen machines. What you have is whatever your system chose to tell you about itself. Observability is the practice of making sure it tells you enough to answer questions you didn’t know you’d need to ask.

Monitoring versus observability

Traditional monitoring watches for failure modes you already predicted: is the disk full, is the process up, is the error rate above a threshold. That remains essential. But distributed systems fail in novel, combinatorial ways, such as one customer, on one app version, hitting one shard during a deploy.

Majors, Fong-Jones and Miranda frame observability as the ability to investigate these unknown unknowns by slicing telemetry in arbitrary ways, without shipping new code first. The data you emit, and whether you kept enough detail, determines whether that’s possible.

The three signal types

Logs, metrics and traces are often called the “three pillars”. They are better understood as three views with different strengths and costs.

Logs: what happened, in detail

A log records a discrete event: a request completed, a payment failed, a retry happened. Logs carry rich context and are the natural place to look when investigating one specific failure. Their weakness is volume and cost, and a pile of text is poor at showing trends.

Metrics: how much, over time

A metric is a number aggregated over time: requests per second, error count, 99th percentile latency, queue depth. Metrics are compact, fast to query and ideal for dashboards and alerting. The price of that compactness is detail: a spike in errors tells you that something is wrong, rarely who is affected or why.

Metric systems also struggle with cardinality. Each unique combination of labels creates its own time series, so labelling by user ID or request ID can explode storage. Unfortunately, those high-cardinality fields are often exactly what isolates a problem.

Traces: where the time went

A distributed trace follows one request through every service it touches. It consists of spans, each a timed unit of work (an HTTP handler, a database query, a cache lookup) with a parent. Laid out as a waterfall, a trace makes it obvious that the slow part of an 8-second checkout was the fraud-check service waiting on a lock.

Google’s Dapper paper popularised this approach, and today OpenTelemetry provides a vendor-neutral standard for emitting all three signals.

Structured logging

Compare these two log lines:

Payment of $49.99 failed for user 812: card declined

{"level":"error","event":"payment_failed","user_id":812,
 "amount_cents":4999,"reason":"card_declined","trace_id":"4bf92f..."}

The first is written for a human reading one line. The second is written for a machine searching millions. With structured logging you can ask “count payment_failed grouped by reason in the last hour” without writing regular expressions that break when someone rewords the message. Including the trace_id lets you jump from a log line to the full trace of that request.

Structured events rich with context are the foundation many observability tools build on. A single wide event per request, with dozens of fields, can often answer questions that would otherwise need separate logs and metrics.

How tracing connects services

Tracing depends on context propagation. When service A calls service B, it passes the trace ID and its current span ID, typically in an HTTP header such as the W3C traceparent header. Service B creates its spans as children of that context and passes it on again.

If any single hop fails to propagate context, the trace fractures into disconnected pieces. That’s why tracing is usually handled by instrumentation libraries rather than by hand. High-traffic systems also sample traces, keeping a fraction, because recording everything is expensive.

SLIs, SLOs and error budgets

Telemetry needs a definition of “good enough”. Google’s SRE practice supplies three terms:

• An SLI (service level indicator) is a measurement of user experience, such as the fraction of requests that succeed in under 300 milliseconds.
• An SLO (service level objective) is a target for that SLI, such as 99.9% over a rolling 30 days.
• An SLA (service level agreement) is a contract with penalties, usually set looser than the internal SLO.

Why not aim for 100%? Because users can’t perceive reliability beyond what their own phone and Wi-Fi provide, each extra “nine” costs far more than the last, and a 100% target makes every deployment unacceptable.

The gap between the SLO and perfection is the error budget. A 99.9% SLO allows 0.1% failures, roughly 43 minutes of total outage in 30 days. While budget remains, the team ships features and takes risks. When it’s exhausted, the priority moves to reliability. This turns the eternal argument between “move fast” and “don’t break things” into a shared, measurable policy.

Alert on symptoms, not causes

Every page should mean a human needs to act now. The SRE book’s guidance is to alert on symptoms users experience, such as elevated errors, high latency or an SLO burning too fast, rather than on causes like high CPU or a nearly full connection pool.

Cause-based alerts fail in both directions. CPU at 90% might be perfectly healthy, so the alert is noise. Meanwhile, a failure mode nobody predicted triggers no cause alert at all, even while users suffer. Symptom alerts catch problems regardless of their origin.

Causes still matter. They belong on dashboards, where they help diagnose a problem a symptom alert has already surfaced.

Noisy alerting has a human cost: engineers learn to ignore pages, and the real one gets missed. A small number of well-chosen, SLO-based alerts beats a hundred thresholds.

Putting it together

A typical investigation uses all three views. A symptom alert fires from a metric. A dashboard narrows the problem to one endpoint. Traces for slow requests show the delay in one downstream service. Logs attached to those traces reveal the specific error. Each signal answers a different question, and the skill is knowing which one to reach for.