What Is Speculative Decoding? LLM Inference Explained

Speculative decoding is an inference technique that speeds up text generation from a large language model by pairing it with a smaller, faster “draft” model. The draft model guesses several tokens ahead, and the large model checks all of those guesses in a single forward pass instead of generating one token at a time. When the guesses are good, you get the large model’s output quality at closer to the small model’s speed.

Why generation is slow to begin with

An LLM generates text autoregressively: it predicts one token, appends it to the sequence, and predicts the next token conditioned on everything so far. Each step requires a full forward pass through the model. For a large model, that forward pass is expensive, and because each token depends on the last, the steps can’t be parallelized — you’re stuck paying that cost, one token at a time, for every single word.

This is fundamentally different from training, where the whole target sequence is known in advance and the model can process it in parallel. At inference time, decoding is sequential by nature, and that sequential bottleneck is a big part of why LLM responses feel slower than a single computation would suggest — it’s many small computations, each waiting on the last.

How the draft-and-verify loop works

Speculative decoding breaks the sequential bottleneck with a two-model pipeline:

1. A small, cheap draft model generates a handful of candidate tokens (say, four or five) autoregressively, just as normal — but because it’s small, this is fast.
2. The large target model then runs a single forward pass over the original sequence plus all the candidate tokens, computing what it would have predicted at each position.
3. The target model’s predictions are compared to the draft’s guesses, token by token, from the start. Every token where they agree is accepted. At the first disagreement, the draft’s guess is discarded, and the target model’s own prediction is used instead — then the process starts over from there.

The key trick is that checking whether a token would have been generated is a parallel operation across all candidate positions, while generating them one at a time is not. Verifying five guesses costs roughly one forward pass of the large model, not five.

Why output quality doesn’t drop

The result is mathematically equivalent to sampling from the large model alone — this isn’t an approximation or a smaller/dumber model standing in for the big one. The target model always has the final say on every token; the draft model only proposes candidates that get checked against what the large model would have generated anyway. When the draft guesses well, you get long runs of accepted tokens for the price of one big forward pass. When it guesses poorly, you fall back to normal token-by-token generation for that stretch, so speculative decoding never makes output worse — only, in the worst case, no faster than the baseline.

Where it fits among other efficiency techniques

Speculative decoding is one of several inference optimizations that stack together rather than compete:

• Quantization shrinks each model’s weights so both the draft and target run cheaper per step.
• Model distillation is often how the draft model itself gets built — a small model trained to imitate the large one’s behavior, which is exactly the property speculative decoding needs.
• Prompt caching avoids recomputing the shared prefix across requests, which is a different bottleneck (input processing) than the one speculative decoding targets (output generation).
• A larger context window increases the cost of each forward pass, which is exactly why cutting the number of large-model passes needed per output token matters more as contexts grow.

Standard decoding vs speculative decoding

	Standard autoregressive decoding	Speculative decoding
Tokens per large-model pass	1	Multiple (when guesses are accepted)
Extra model required	No	Yes, a draft model
Output distribution	Baseline	Identical to baseline
Best case speedup	—	Higher acceptance rate → faster
Worst case	Baseline speed	Roughly baseline speed
Extra memory/compute overhead	None	Running two models

Where it helps most — and where it doesn’t

Speculative decoding shines when the draft model’s guesses are frequently correct, which tends to happen with predictable text — boilerplate code, common phrasing, repeated structure. It helps less on highly creative or unpredictable output, where the draft model’s guesses are rejected often and the extra verification overhead buys little. It also requires having a suitable draft model on hand: one small and fast enough to be worth running, but similar enough in behavior to the target model that its guesses land often. This is why it’s typically deployed by the same team or provider that serves the large model, rather than something an application developer configures directly — it’s an inference-serving optimization, not an API parameter. If you’re comparing the economics of running models at scale, our LLM token cost calculator is a useful companion for thinking through the latency and cost tradeoffs involved.

The takeaway

Speculative decoding speeds up LLM inference by letting a small draft model propose several tokens ahead, then having the large model verify them all in one parallel pass instead of generating each one sequentially. Accepted tokens are free speed; rejected tokens fall back to normal generation, so quality never suffers. It’s one of several complementary techniques — alongside quantization, distillation, and caching — that inference providers use to make large models feel faster without changing what they actually produce.