What Is a KV Cache? Why LLM Inference Speeds Up

A KV cache (key-value cache) is memory an LLM keeps during text generation to avoid recomputing the attention calculations for tokens it has already processed. Without it, generating each new token would require redoing the full attention computation over the entire preceding sequence from scratch. With it, that earlier work is stored once and reused, which is why generation gets dramatically cheaper per token once the cache is warm — and why it’s also one of the biggest consumers of GPU memory during inference.

Where “key” and “value” come from

Every transformer layer’s self-attention mechanism projects each token into three vectors: a query, a key, and a value. Roughly, the query asks “what am I looking for,” the key answers “what do I represent,” and the value carries “what information do I contribute if selected.” Attention scores are computed by comparing a token’s query against every other token’s key, and the result is used to weight a combination of every token’s value.

The critical detail: for a given input sequence, a token’s key and value vectors don’t change as generation proceeds — only new tokens add new keys and values. The query for the current token being generated is the only thing that’s new at each step.

What gets cached, and why it helps

Since keys and values for already-processed tokens are fixed, there’s no reason to recompute them every time a new token is generated. The KV cache stores every previous token’s key and value vectors, per layer, so that generating token N+1 only requires computing the query, key, and value for token N+1, then attending against the cached keys and values for tokens 1 through N.

Without caching, generating a response of length N would redo an amount of computation that grows roughly with the square of the sequence length, since each new token would reprocess the entire sequence so far. With caching, each new token’s cost stays roughly proportional to the current sequence length rather than to redoing work for every prior token — a substantial difference for anything beyond very short outputs.

The tradeoff: memory instead of compute

The KV cache isn’t free — it trades recomputation for memory. Its size scales with sequence length, number of layers, number of attention heads, and the dimension of each head, all multiplied by two (for keys and values). For long context windows, the KV cache can end up consuming more GPU memory than the model’s weights themselves, which is a major reason serving very long contexts is expensive and why providers often charge differently for long-context requests — see our guide to prompt caching and LLM costs for how providers pass some of this savings back when a prompt prefix repeats across requests.

This memory pressure is also why techniques like multi-query attention and grouped-query attention exist: by having multiple query heads share the same key/value heads, the cache shrinks substantially, at some cost to model quality, trading a bit of capacity for a lot less memory per token of context.

Why this matters for batching and latency

Because the KV cache is per-request, serving many concurrent conversations means maintaining a separate cache for each one, all resident in GPU memory simultaneously. This is a central constraint behind how inference servers decide how many requests to batch together — see batch vs real-time inference for the broader tradeoff. A request with a very long conversation history occupies proportionally more cache memory than a short one, which is part of why some providers cap context length more aggressively under load, and why techniques like cache eviction or offloading older entries to slower memory show up in production inference stacks.

The KV cache is also why the first token of a response typically takes noticeably longer to produce than subsequent ones: the first token requires populating the cache for the entire input prompt (the “prefill” phase), while every token after that reuses it (the “decode” phase). Providers sometimes report these as separate latency numbers — time-to-first-token and per-token latency — precisely because they’re governed by different bottlenecks.

KV caching vs prompt caching

These two are related but not identical. KV caching is an inference-time mechanism within a single request (or a session an inference server keeps warm), holding computed attention state so it isn’t redone token by token. Prompt caching, as offered by several API providers, is a higher-level feature that reuses a prefix’s KV cache across separate API calls that share the same starting prompt — a system prompt or a long document, for instance — so repeated calls with the same prefix skip the prefill cost entirely. Prompt caching is, in effect, KV caching deliberately kept alive and shared across requests rather than discarded once a single generation finishes. If you’re estimating request costs at scale, our LLM token cost calculator can help translate these savings into dollar terms.

This also connects to why speculative decoding is a separate optimization rather than a competing one: it changes how many tokens get verified per step, while the KV cache is what makes each individual step cheap regardless of that strategy.

The takeaway

A KV cache stores each token’s attention key and value vectors so an LLM doesn’t recompute them for every subsequent token, turning what would be roughly quadratic generation cost into something closer to linear. The cost of that speedup is memory — the cache grows with context length and can dwarf the model weights themselves at long context lengths, which is why it drives so much of the engineering behind serving LLMs cheaply and at low latency.