Distributed Tracing Explained: Following Requests Across Services

Distributed tracing is the practice of following a single request as it moves through every service it touches, stitching each hop into one end-to-end timeline. In a microservices architecture, one user action — say, loading a checkout page — might fan out across a dozen internal calls: an API gateway, an auth service, an inventory service, a payments service, a database or two. When that request is slow or fails, distributed tracing is what tells you which hop was the problem, not just that the overall request was slow.

Why logs and metrics aren’t enough

Logs, metrics, and traces each answer a different question, and traces exist because the other two run out of answers at the boundary of a single service. Metrics tell you that p99 latency spiked. Logs from an individual service tell you what happened inside that service. Neither one, alone, tells you the causal chain across service boundaries — that request A called service B, which called service C, which was the one actually slow. Reconstructing that chain from a pile of independent log lines, each service’s clock slightly out of sync with the others, is close to impossible at scale. Tracing solves it by design, not by correlation after the fact.

Traces, spans, and context propagation

A trace represents one request’s full journey. It’s made up of spans — each span is one unit of work, typically one service call or one function boundary, with a start time, a duration, and metadata (status code, error flags, custom tags). Spans are organized as a tree: a parent span for the incoming request, child spans for everything that request triggered downstream.

The mechanism that ties spans together across service boundaries is context propagation. When service A calls service B, it attaches a trace ID and the current span ID to the outgoing request — usually in HTTP headers. Service B reads those headers, creates its own span as a child of A’s span, and propagates the same trace ID onward to whatever it calls next. Every service in the chain has to participate for the trace to stay unbroken; a single service that drops the header creates a gap in the trace.

What a trace actually shows you

Visualized, a trace typically renders as a waterfall or flame graph: each span drawn as a horizontal bar, nested under its parent, positioned and sized by start time and duration. This view answers questions that are hard to get any other way:

• Which service in the chain accounted for most of the total latency
• Whether calls happened sequentially when they could have run in parallel
• Where retries or duplicate calls are quietly adding latency
• Which specific downstream dependency caused a cascading failure

A trace that shows one span consuming 800ms of a 900ms total request immediately narrows an investigation that log-grepping across ten services would otherwise take an hour.

Sampling: why you don’t trace everything

Tracing every single request in a high-traffic system generates enormous data volume and per-request overhead. Most tracing systems apply sampling — capturing a percentage of traces (head-based sampling, decided at the start of a request) or capturing based on outcome (tail-based sampling, keeping traces that were slow or errored, decided after the fact). Tail-based sampling is more useful for debugging since it preferentially keeps the interesting traces, but it requires buffering spans until the request completes before deciding whether to keep them, which costs more infrastructure than head-based sampling.

Distributed tracing vs related observability tools

	Distributed tracing	Log aggregation	Metrics dashboards
Answers	Where did time go across services?	What happened in this service?	Is the system healthy overall?
Granularity	Per-request, cross-service	Per-event, single service	Aggregated over time
Best for	Latency debugging, root cause across hops	Detailed forensics within a service	Alerting, trend detection
Data volume	High (often sampled)	Very high	Low (pre-aggregated)

These tools complement rather than replace each other. A metrics dashboard built on SLIs and SLOs tells you an alert fired; a trace tells you which service caused it; aggregated logs from that specific service give you the exact error. Mature observability setups correlate all three, often letting you jump from a slow span directly to the logs emitted during that span.

Where tracing fits in a service mesh

In architectures built around a service mesh or an API gateway, a meaningful share of trace instrumentation can happen at the infrastructure layer rather than in application code — the mesh’s sidecar proxies can generate spans for every hop automatically, since they already sit in the request path. Application-level instrumentation is still usually needed for spans inside a service (a slow database query, a specific function), but the cross-service propagation and much of the span creation can be handled without touching business logic.

The takeaway

Distributed tracing reconstructs the true path of a request across service boundaries by tying together spans with a shared trace ID, propagated hop by hop. It’s the tool that answers “which service actually caused this” in an architecture where a single user action can touch a dozen independent services — a question logs and metrics, used alone, structurally can’t answer. Sampling keeps the overhead manageable; correlating traces with logs and metrics is what turns a slow request from a mystery into a five-minute diagnosis.