LLM Observability: Tracing AI Agents in Production

LLM observability is the practice of capturing, in structured detail, everything that happens during a call to a language model or an AI agent’s run — the exact prompt sent, the model’s response, every tool call made along the way, how many tokens were consumed, and how long each step took — so that a slow, wrong, or expensive result can actually be traced back to its cause instead of just observed as a black box.

Why ordinary observability doesn’t cover it

Logs, metrics, and traces already form the backbone of conventional observability: a trace follows a request through a distributed system’s services, a metric aggregates a number over time, a log records a discrete event. That toolkit still applies to an LLM-backed system, but it misses what actually goes wrong in one. A conventional trace shows that a request took 4 seconds and hit three services; it doesn’t show what the model was actually asked, what it decided to do with an ambiguous instruction, or why it called the wrong tool. The failure modes are semantic, not just latency or error codes — a request can return a 200 and still be a bad answer.

This is the gap LLM observability closes: it treats the content of a prompt and response as first-class data to capture, not just the fact that a call happened.

What actually gets traced

A single user request to an agent can fan out into dozens of underlying calls — a router deciding which sub-agent should handle the query, a retrieval step, one or more tool invocations, and a final generation step that assembles everything into an answer. LLM observability tooling typically captures, for every one of those steps:

• The full prompt and completion, including the system prompt, any injected context, and the model’s raw output before post-processing.
• Tool calls and their arguments, along with what the tool actually returned — essential for debugging a multi-agent system where one agent’s bad output becomes the next agent’s bad input.
• Token counts and cost, per call and rolled up per request, which is what turns a vague sense that “the agent got expensive” into a specific line showing which step burned the tokens. A tool like a token cost calculator is useful for estimating cost up front; a trace is what shows where it actually went after the fact.
• Latency per step, distinguishing model inference time from tool execution time from network overhead, since the fix for each is completely different.
• Model and parameter metadata — which model version, what temperature, what sampling settings — since the same prompt can behave differently across a model update.

Strung together, these steps form a trace tree much like a distributed trace, except the spans carry prompt and completion text instead of just service names and durations.

Tracing versus evaluation

Tracing and evaluation solve related but distinct problems, and it’s worth keeping them separate. A trace answers “what actually happened on this specific request” — it’s inherently retrospective and per-instance. An LLM eval answers “how good is this prompt or model, on average, across many cases” — it’s aggregate and often run before a change ships, not just after. In practice the two feed each other: a trace surfaces a concrete failure — the agent hallucinated a fact, called the wrong tool, or looped — and that failure becomes a new test case added to an eval suite, so a regression on that exact scenario gets caught automatically the next time a prompt or model changes. Observability without evals catches problems only after they’ve already happened to a real user; evals without observability tend to test scenarios someone thought of in advance rather than the ones that actually occur in production.

Catching the failure modes unique to LLMs

Some of what LLM observability exists to catch has no analog in conventional service monitoring:

• Hallucination. A trace showing the exact context the model had access to makes it possible to tell whether a wrong answer came from missing context or from the model inventing something despite having the right information in front of it — a distinction covered in more depth in why LLMs hallucinate.
• Prompt drift. When a shared system prompt is edited for one use case, tracing across many requests can reveal it quietly degraded behavior on a different use case that depends on the same prompt.
• Tool-call loops. An agent that repeatedly calls the same tool with slightly different arguments, never reaching a final answer, shows up clearly as a repeating pattern in a trace tree — much harder to spot from an aggregate latency metric alone.
• Context window pressure. As conversation history or retrieved context grows, a trace showing the actual token count sent per call makes it visible when a request is approaching a model’s context limit well before it starts silently truncating older context.

Where this fits operationally

LLM observability tooling generally sits alongside, not instead of, standard infrastructure monitoring. The distributed trace for a request still matters for understanding which service was slow; the LLM-specific trace layered on top explains why the model step of that request produced what it did. Cost tracking is often the first thing teams wire up, since prompt caching and similar optimizations are only possible to reason about once there’s visibility into exactly which calls are expensive and why. From there, most teams add automatic flagging — a run that used an unusual number of tool calls, took unusually long, or scored poorly on an automated eval gets surfaced for review rather than requiring someone to read every trace by hand.

The takeaway

LLM observability extends ordinary tracing, logging, and metrics with the one thing generic infrastructure monitoring was never built to capture: the actual content of prompts, completions, and tool calls, and what they cost in tokens and time. A request can succeed by every conventional measure — no errors, reasonable latency — and still be a bad answer, which is precisely the class of failure a full trace of the model’s reasoning and tool use is built to catch. Pair it with a running eval suite fed by the failures traces surface, and an agent’s behavior stops being something you find out about from users and starts being something you can actually debug.