What Is Logit Bias? Steering LLM Output Per Token

Logit bias is a number you add to (or subtract from) a specific token’s score before a language model converts its raw predictions into probabilities and samples the next word. It’s a low-level lever for controlling output that works underneath the prompt, rather than through it — instead of asking the model nicely not to say a word, you make that word mathematically less likely to ever get picked.

To use it, you need to know what a logit actually is.

Logits, before they become probabilities

At each generation step, an LLM produces one raw score — a logit — for every token in its vocabulary, tens of thousands of numbers representing how strongly the model favors each possible next token. Those raw scores aren’t probabilities yet: they can be negative, they don’t sum to 1, and they’re not directly comparable to a percentage chance.

A softmax function converts the logits into an actual probability distribution, and then a sampling strategy — covered in our guide to temperature, top-p, and top-k — picks a token from that distribution. Temperature reshapes the whole distribution’s spread; top-p and top-k cut off which tokens are eligible; logit bias intervenes one step earlier, editing individual scores before any of that happens.

raw logits → [+ logit bias] → softmax → probability distribution → sampling → token

Because the adjustment happens before softmax, a large enough negative bias can push a token’s probability effectively to zero — softmax exaggerates gaps between large negative numbers and everything else — while a large positive bias can make a token dominate the distribution regardless of what the model would otherwise have predicted.

What it’s actually used for

• Banning specific tokens. Set a strongly negative bias on the token IDs for a word you never want output — a competitor’s name, a profanity, a placeholder string that leaked into training data.
• Forcing a token to appear. A strong positive bias makes a token overwhelmingly likely, useful for constraining output into a narrow format alongside structured output modes.
• Nudging without forcing. A small positive or negative bias shifts preference — mildly discouraging a filler word, or mildly encouraging a particular phrasing — without eliminating alternatives entirely.
• Classification shortcuts. Bias the tokens for “yes”/“no” or a small fixed label set upward relative to everything else, effectively turning free-text generation into a constrained choice.

Why not just prompt it instead?

A prompt instruction like “never say the word X” competes with everything else in the context window for the model’s attention, and models are probabilistic — an instruction lowers a token’s likelihood, but doesn’t guarantee it never appears, particularly in a long response where the constraint drifts out of focus. It’s also indirect: the model has to interpret the instruction correctly before it can act on it, and interpretation is exactly the part that’s inconsistent.

Logit bias operates below the level of interpretation. It doesn’t matter whether the model “understands” the constraint — the token’s score is adjusted before the model gets a chance to consider it at all. That makes it a better fit for hard constraints (a token that must never or must always appear) than a soft stylistic preference, which is still better handled through the prompt or a system prompt.

The tradeoff is granularity: logit bias operates on tokens, not words or concepts, and a single word can span multiple tokens depending on the tokenizer. Banning a word reliably sometimes means biasing several different token sequences that could produce it — a common word, its capitalized form, and any subword pieces that could be recombined to spell it.

Logit bias vs sampling parameters vs structured output

Technique	Operates on	Best for
Logit bias	Individual token scores	Hard bans/forces on specific tokens
Temperature / top-p / top-k	The whole probability distribution	General creativity vs. determinism
Structured output (JSON mode)	The full output grammar	Enforcing valid syntax, not word choice
Prompt instructions	Model interpretation	Soft, contextual, negotiable constraints

These aren’t mutually exclusive — a production pipeline might use structured output to guarantee valid JSON, temperature to control how varied the free-text fields are, and logit bias to hard-ban a handful of forbidden tokens inside those fields, all at once.

Practical limits

Not every API surface exposes logit bias, and where it exists, it’s typically capped to a fixed number of token IDs per request rather than an open-ended list — providers cap it because applying a bias to every possible token would be both slow and rarely useful. It also requires knowing token IDs ahead of time, which means running the target words through the same tokenizer the model uses, since a word’s token breakdown isn’t always what a human would guess from spelling alone.

Because it works at the token level, logit bias also can’t reliably suppress a concept that can be expressed many different ways — banning the tokens for “cheap” doesn’t stop the model from writing “inexpensive,” “budget,” or “low-cost” instead. For concept-level control, prompt-based guidance or output filtering after generation are usually the better tool, even though they’re less deterministic than a direct token edit.

The takeaway

Logit bias adjusts a model’s raw token scores before sampling, giving you deterministic control over specific tokens that a prompt instruction can only approximate. It’s the right tool for hard bans and forced tokens at the vocabulary level — not for steering tone, style, or ideas that can be phrased many ways, where prompting and structured output remain the better fit.