What Is LLM Temperature? Sampling Parameters Explained

Temperature is a parameter that controls how random or deterministic a language model’s output is, by reshaping the probability distribution it samples the next token from. Low temperature makes the model favor its highest-probability tokens, producing focused, repeatable output. High temperature flattens the distribution, giving lower-probability tokens a real chance of being picked, which produces more varied and sometimes more surprising output.

It’s one of the first knobs most people encounter when calling an LLM API, and one of the most misunderstood — it doesn’t control “creativity” directly, it controls randomness, and those aren’t quite the same thing.

How token generation actually works

An LLM generates text one token at a time. At each step, the model doesn’t just output a token — it outputs a probability distribution over its entire vocabulary, scoring how likely each possible next token is given everything generated so far. Some sampling strategy then has to turn that distribution into an actual choice.

The simplest strategy, greedy decoding, always picks the single highest-probability token. It’s fully deterministic but tends to produce dull, repetitive text, and can get stuck in loops. Temperature is what lets you move away from pure greedy decoding toward sampling — picking a token probabilistically according to the distribution, rather than always taking the top one.

What temperature actually does to the distribution

Temperature is applied before the probabilities are computed, by dividing the model’s raw output scores (logits) by the temperature value and then converting the result into probabilities. The effect:

• Temperature near 0 sharpens the distribution toward the single most likely token — output becomes close to deterministic, and running the same prompt twice tends to produce nearly identical results.
• Temperature around 1 leaves the distribution close to what the model naturally predicts — this is usually treated as a neutral default.
• Temperature above 1 flattens the distribution further, giving less likely tokens more relative weight. Output becomes more varied, but past a point it starts to include tokens that don’t make sense in context, and coherence degrades.

The key thing to understand is that temperature doesn’t add new information or “creativity” to the model — it only reweights probabilities the model already computed. A high temperature can’t make a model produce an idea it wasn’t already capable of generating; it just makes unlikely (from the model’s perspective) continuations more reachable.

Top-p and top-k: the other sampling controls

Temperature is usually paired with one or both of these:

• Top-k sampling restricts the choice to the k highest-probability tokens, discarding the long tail entirely before sampling. A top-k of 40 means only the 40 most likely next tokens are even considered.
• Top-p sampling (also called nucleus sampling) instead keeps the smallest set of tokens whose cumulative probability exceeds p — so the cutoff adapts to how confident the model is at each step, rather than using a fixed count.

These parameters constrain which tokens are eligible before temperature decides how randomly to pick among them. A common combination is a moderate temperature with top-p around 0.9, which lets the model sample from a reasonably wide but not unbounded set of plausible tokens.

Picking a value for your use case

There’s no single correct temperature — it depends on what you’re generating:

• Low (near 0 to ~0.3) — code generation, data extraction, factual Q&A, anything with a single correct or preferred answer, and anywhere you want function calling arguments to come out consistently formatted.
• Moderate (~0.5 to 0.8) — general conversation, summarization, and most everyday assistant use cases, where some variation in phrasing is fine but you still want coherent, on-topic output.
• High (~0.9 and up) — brainstorming, creative writing, or generating multiple diverse options to choose from, where variety matters more than any single output being “the best” one.

Reasoning-heavy tasks are a special case: many reasoning models are tuned to work best at a specific default temperature, and pushing it too far in either direction can hurt the reliability of their intermediate reasoning steps, so it’s worth checking a given model’s recommended range rather than assuming general guidance applies unchanged.

Reproducibility and cost

Because temperature introduces randomness, two identical calls with temperature above 0 can return different output even with the same prompt — which matters for testing and debugging. Some APIs offer a seed parameter alongside temperature to make sampling more reproducible for a fixed model version, though determinism still isn’t always guaranteed across model updates.

Temperature itself doesn’t change how many tokens you use, so it has no direct effect on API cost — the token cost calculator tool estimates spend based on input and output token counts regardless of the sampling settings you choose.

The takeaway

Temperature reshapes the probability distribution an LLM samples its next token from — low values push toward the most likely, deterministic continuation, and high values flatten the distribution toward more varied, less predictable output. It’s usually combined with top-p or top-k to bound which tokens are even eligible before sampling. Pick a low temperature when you need consistency and correctness, and a higher one when you want variety — and expect to tune it per task rather than treating one value as universally right.