Temperature vs Top-p vs Top-k: LLM Sampling Explained

Temperature, top-p, and top-k are the three parameters that control how an LLM turns its raw probability predictions into an actual chosen word. At each step, the model doesn’t output one guaranteed next token — it outputs a probability distribution over its entire vocabulary, and a sampling strategy decides which token actually gets picked. These three settings shape that decision differently, and mixing them up is a common source of “why does my API output look so different from the playground” confusion.

Why sampling exists at all

An LLM’s final layer produces a score, called a logit, for every token in its vocabulary — tens of thousands of candidates. A softmax function converts those logits into probabilities that sum to 1. The simplest strategy, greedy decoding, always picks the single highest-probability token. It’s deterministic, but it tends to produce flat, repetitive text, because the model keeps taking the “safest” path at every single step instead of ever taking a slightly-less-likely but more interesting one.

Sampling introduces controlled randomness into that choice, and temperature, top-p, and top-k are the three most common ways to control how much randomness — and where it’s allowed to come from.

Temperature: reshaping the whole distribution

Temperature rescales the logits before the softmax step, controlling how sharp or flat the resulting probability distribution is.

• Low temperature (near 0) sharpens the distribution — the highest-probability token dominates even more, pushing the output toward greedy, deterministic behavior.
• Temperature of 1 leaves the distribution roughly as the model produced it.
• High temperature (above 1) flattens the distribution, giving lower-probability tokens a meaningfully better chance of being picked, which increases variety at the cost of coherence.

Temperature affects every token in the vocabulary proportionally — it doesn’t remove any candidates, it just changes how evenly probability mass is spread across all of them. This is a different lever from what we cover in our temperature-specific explainer, which goes deeper into the math; this piece is about how temperature interacts with the other two knobs.

Top-k: a hard cutoff by rank

Top-k sampling truncates the candidate pool to the k highest-probability tokens, then renormalizes and samples only from that fixed-size shortlist. If k = 40, the model considers exactly the 40 most likely next tokens, regardless of how the probability mass is actually distributed among them.

The weakness of top-k is that it’s a fixed count, not a fixed confidence level. When the model is very confident (one token dominates), a top-k of 40 still includes 39 tokens that shouldn’t realistically be considered. When the model is genuinely uncertain across many plausible tokens, top-k might cut off legitimate candidates that fell just outside the arbitrary rank cutoff.

Top-p: a dynamic cutoff by probability mass

Top-p sampling (also called nucleus sampling) fixes that weakness by choosing a cutoff based on cumulative probability rather than a fixed count. With top_p = 0.9, the model includes the smallest set of highest-probability tokens whose probabilities sum to at least 90%, then samples only from that set.

This means the candidate pool automatically shrinks when the model is confident (a handful of tokens might already cover 90% of the mass) and automatically grows when the model is uncertain (many tokens are needed to reach 90%). That adaptiveness is why top-p is the more commonly used default in most modern APIs.

How they interact

Most APIs let you set temperature and one of top-p or top-k simultaneously, and the order of operations matters: typically the vocabulary is filtered first (by top-k or top-p), and temperature is applied to reshape the probabilities of whatever survives the filter, before the final sample is drawn. That means:

• A very low top-p or top-k with a high temperature still can’t produce wild outputs, because the candidate pool was already narrowed before temperature had anything to reshape.
• A high top-p or top-k with a low temperature will look nearly deterministic anyway, because temperature has already sharpened the distribution toward one dominant token before sampling happens.

Parameter	Controls	Cutoff type	Typical default range
Temperature	Sharpness of the whole distribution	None — reshapes, doesn’t remove	0.0–1.0
Top-k	Candidate pool size	Fixed count	20–50
Top-p	Candidate pool size	Cumulative probability	0.9–0.95

Practical guidance

• Deterministic, factual tasks (code generation, data extraction, following chain-of-thought prompts precisely) favor low temperature and either a tight top-p (around 0.9) or no top-k restriction at all, since the model’s own confidence is usually already well-calibrated for these tasks.
• Creative or exploratory tasks (brainstorming, fiction, varied phrasing) benefit from higher temperature and a looser top-p, giving the model more room to deviate from its single most likely continuation.
• Reasoning-heavy tasks, including the extended reasoning traces used by reasoning models, often use lower temperature specifically because compounding randomness across a long chain of intermediate steps increases the odds that one bad token derails the entire chain.

Changing these settings doesn’t change how many tokens a request costs — token usage is about counting, not sampling — but if you’re iterating on sampling settings through repeated API calls, our LLM token cost calculator is useful for estimating what that experimentation will cost you at scale.

The takeaway

Temperature reshapes the probability distribution across the entire vocabulary; top-k and top-p both truncate the candidate pool before sampling, but top-k uses a fixed count while top-p uses a dynamic, confidence-aware cumulative threshold. In most APIs, filtering happens first and temperature reshapes what’s left, so the two settings interact rather than acting independently. Start from a provider’s defaults, and adjust one parameter at a time — changing temperature and top-p together makes it much harder to tell which knob actually caused a change in output.