In-Context Learning vs Fine-Tuning for LLMs

In-context learning and fine-tuning are two different ways to get a large language model to perform a task it wasn’t explicitly trained to do out of the box. In-context learning shows the model examples inside the prompt itself, at inference time, with no change to the model’s underlying weights. Fine-tuning updates the weights directly through additional training, permanently changing how the model behaves on every future request.

In-context learning: teaching through the prompt

When you include examples directly in a prompt — “here are three sentences and their sentiment labels, now classify this fourth one” — you’re using in-context learning. The model has never been trained specifically on your labeled examples; it’s using patterns learned during its original pretraining to recognize the task you’re demonstrating and generalize from your examples to the new input.

This is often broken down by how many examples you provide: zero-shot (no examples, just an instruction), one-shot (a single example), and few-shot (a handful of examples) — see zero-shot vs few-shot prompting for how the tradeoffs shift as you add more examples. Chain-of-thought prompting is a related in-context technique — rather than demonstrating input-output pairs, you demonstrate the reasoning steps that lead to an answer.

The mechanism behind why this works is still an area of active research, but the practical effect is well established: transformer-based models can pick up a pattern from a handful of prompt examples and apply it to new inputs, without any weight updates at all.

Fine-tuning: updating the weights

Fine-tuning takes a pretrained model and continues training it on a task-specific dataset, adjusting the model’s weights through gradient descent so the behavior you want becomes baked into the model itself, rather than something you have to re-demonstrate in every prompt. Once fine-tuned, the model performs the target task without needing any examples in the prompt at inference time — the “learning” happened during training, not at request time.

Full fine-tuning updates all of a model’s parameters, which is expensive and requires meaningful compute and data. LoRA fine-tuning is a cheaper alternative that freezes most of the original weights and trains small additional low-rank matrices instead, achieving much of the benefit of full fine-tuning at a fraction of the compute and storage cost.

Comparison table

	In-context learning	Fine-tuning
When it happens	At inference time, per request	During a separate training step, once
Changes model weights	No	Yes
Setup cost	None — just write examples into the prompt	Requires a labeled dataset and training run
Per-request cost	Higher — examples consume prompt tokens every call	Lower — no extra tokens needed after training
Flexibility	Easy to change task by editing the prompt	Requires retraining to change behavior
Best for	Quick iteration, tasks that shift often	Stable, high-volume, well-defined tasks

The cost tradeoff

In-context learning has effectively zero setup cost — you can change the task by editing the prompt and get a result immediately — but every example you include consumes prompt tokens on every single request. For a high-volume application making millions of calls, those repeated example tokens add up; a token cost calculator makes the arithmetic concrete when comparing a longer few-shot prompt against a fine-tuned model that needs no examples at all. Prompt caching can offset some of this if the same examples are reused across many requests, since a cached prefix doesn’t need to be reprocessed from scratch each time.

Fine-tuning flips this: the upfront cost is higher (you need a dataset and a training run), but the per-request cost drops, since a fine-tuned model doesn’t need lengthy examples in every prompt to behave correctly.

When retrieval changes the calculus

Neither technique is the right tool when the task requires up-to-date or proprietary information the model was never trained on — that’s what retrieval-augmented generation is for, fetching relevant documents at request time and inserting them into the prompt. RAG and in-context learning often work together: the retrieved documents become part of the context, and the model uses in-context learning-style pattern matching to answer based on them. Fine-tuning, by contrast, can’t inject facts that change day to day — it’s suited to teaching a stable skill or format, not to keeping pace with a changing knowledge base.

Which to choose

In-context learning is usually the right starting point: it’s fast to iterate on, requires no training infrastructure, and lets you validate whether a task is even solvable by the model before investing in fine-tuning. Move to fine-tuning when the task is stable, well-defined, high-volume enough that per-request token costs matter, or when you need behavior — tone, format, a narrow domain vocabulary — that’s hard to fully specify through examples alone, no matter how many you include in the prompt.

The takeaway

In-context learning teaches a model a task through examples placed directly in the prompt, with no change to the model itself — flexible and instant, but costing tokens on every request. Fine-tuning updates the model’s weights through additional training, trading upfront setup cost for cheaper, more consistent behavior at inference time. Most teams start with in-context learning to validate an approach, then fine-tune once the task, volume, and cost profile justify the investment.