Chain-of-Thought Prompting Explained

Chain-of-thought prompting is a technique that asks a large language model to work through a problem step by step, in visible text, before giving a final answer — rather than jumping straight to a conclusion. The core idea is straightforward: many problems that a model gets wrong when asked to answer directly, it gets right when asked to first reason through the intermediate steps out loud.

Why intermediate steps help

An LLM generates text one token at a time, and each token it produces becomes part of the context it uses to generate the next one. When a model is asked to answer a multi-step math or logic problem directly, it has to arrive at the final answer in effectively one shot, with no scratch space to work through the intermediate reasoning. When it’s prompted to reason step by step instead, each intermediate step it writes becomes additional context available for producing the next step — the model is, in effect, using its own output as working memory.

This matters most on tasks that genuinely require sequential reasoning — arithmetic word problems, multi-step logic puzzles, or questions with several dependent sub-parts. For simple factual lookups or tasks with no real reasoning chain, chain-of-thought tends to add length without adding accuracy.

A basic example

Direct prompting:

Q: A store had 23 apples. It sold 15 and received a shipment of 8 more. How many apples does it have now? A: 16

Chain-of-thought prompting:

Q: A store had 23 apples. It sold 15 and received a shipment of 8 more. How many apples does it have now? Think step by step. A: Start with 23 apples. After selling 15, there are 23 − 15 = 8 apples. After receiving 8 more, there are 8 + 8 = 16 apples.

Both examples land on the same answer here, but on harder or longer problems, forcing the intermediate steps into the open substantially reduces arithmetic slips and dropped conditions that direct prompting is prone to.

Zero-shot vs few-shot chain-of-thought

There are two common ways to elicit this behavior:

• Zero-shot chain-of-thought — simply appending an instruction like “think step by step” or “explain your reasoning” to the prompt, with no examples provided.
• Few-shot chain-of-thought — including one or more worked examples in the prompt that demonstrate the step-by-step reasoning pattern you want the model to follow, before asking the actual question.

Few-shot examples tend to produce more consistent formatting and reasoning style, since the model has a concrete pattern to imitate rather than inferring what “step by step” should look like. The tradeoff is prompt length — each worked example consumes context and, for API usage, tokens you’re paying for on every request.

Chain-of-thought vs reasoning models

It’s worth distinguishing prompted chain-of-thought from models purpose-built for extended reasoning. Chain-of-thought prompting is a technique you apply to any capable model through the prompt alone. Reasoning models, by contrast, are trained specifically to generate extended internal reasoning before answering, often as a distinct mode or step separate from the final response, and typically do this by default on harder queries without needing an explicit “think step by step” instruction. In practice, the two approaches overlap: even reasoning models can benefit from a well-structured prompt, and chain-of-thought prompting on a non-reasoning model can recover some — though usually not all — of the accuracy gains that a dedicated reasoning model achieves natively.

Where it fits in an application

If you’re building on top of an LLM through an API, chain-of-thought is one of the cheapest accuracy levers available — it costs a longer prompt and a longer response, not a different model or additional infrastructure. A few practical notes:

• It increases output length, which increases both latency and cost, so it’s worth reserving for problems that actually benefit rather than applying it universally. See prompt caching for one way to offset the token cost of repeated few-shot examples across many requests.
• If your application only needs the final answer, not the reasoning, you can ask the model to keep the reasoning separate from a clearly marked final answer, so you can programmatically extract just the conclusion.
• Chain-of-thought reasoning is still generated by the same underlying model and can still be wrong — it makes errors more inspectable, not impossible. Treat visible reasoning as a debugging aid and an accuracy improvement, not a guarantee of correctness.

Chain-of-thought prompting is also a natural complement to function calling and retrieval-augmented generation in agentic setups — reasoning through what information is needed, or which tool to call and why, before acting tends to reduce mistakes in the same way it does for math problems, since the model is deciding based on articulated intermediate reasoning rather than jumping straight to an action.

The takeaway

Chain-of-thought prompting improves LLM accuracy on multi-step problems by asking the model to reason through intermediate steps in visible text rather than answering directly, effectively giving the model scratch space it can build on token by token. It’s cheap to try — a prompt change, not an infrastructure change — and most valuable on genuinely sequential problems like math, logic, and multi-step decisions, with diminishing returns on simple factual questions that don’t benefit from a reasoning chain.