What Is Model Distillation? Smaller Models, Explained

Model distillation — also called knowledge distillation — is a technique for compressing a large, capable model into a smaller, cheaper one by training the small model to imitate the big one. The large model is the teacher; the small model is the student. Instead of learning from raw labeled data alone, the student learns from the teacher’s outputs — and it turns out a student taught this way ends up meaningfully better than the same-sized model trained from scratch. Distillation is the main reason the fast, cheap tier of every major LLM lineup performs far above its weight.

The teacher-student setup

The recipe has three ingredients: a trained teacher model, a smaller student architecture, and a pile of inputs. You run the inputs through the teacher, record what it produces, and train the student to reproduce those outputs. The teacher never changes; it acts as a data generator and grading key. The student’s job is not to rediscover everything the teacher learned from trillions of tokens — only to approximate the function the teacher computes, which is a much easier target.

Dark knowledge: the classic method

The founding formulation comes from Hinton, Vinyals, and Dean’s 2015 paper. Their insight: a classifier’s wrong answers carry information. When an image model says “90% dog, 7% wolf, 0.001% car,” the relative probabilities among the losing classes — dog is much more like wolf than like car — encode the similarity structure the teacher has learned. Hinton called this dark knowledge.

Classic distillation therefore trains the student on the teacher’s full probability distribution (soft labels) rather than just the single correct answer (hard labels), typically with a temperature parameter that softens the distribution to expose more of that structure. The student gets a far richer training signal per example than ground-truth labels provide.

The canonical early success was DistilBERT (2019): a distilled version of BERT with 40% fewer parameters that kept about 97% of BERT’s language-understanding performance while running 60% faster. Those numbers made the pitch concrete — give up almost nothing, get a model nearly half the size.

How LLM-era distillation differs

With modern generative models, distillation usually looks less like matching probability distributions and more like synthetic-data generation. The teacher writes: answers, explanations, code, step-by-step reasoning. The student is then fine-tuned on those teacher-written examples as ordinary training text. This is often called response-based distillation, and it dominates for two practical reasons:

• Logits are often unavailable. If the teacher is accessed through an API, you get its text output, not its internal probability distributions — so soft-label distillation isn’t even an option.
• It targets behavior, not just predictions. Training on the teacher’s worked examples transfers style, formatting, instruction-following, and reasoning patterns — the things that make a model useful, not merely accurate.

Labs distill internally too, where they do have the logits, and can combine both signals. Either way, the pattern is the same: the flagship model is expensive to run, so its capabilities are pressed down into small models that serve the high-volume, latency-sensitive tiers. When a provider ships a “mini” or “flash” model that punches above its parameter count, distillation from the flagship is almost always part of the story. It also complements sparse architectures: a giant Mixture of Experts teacher can be distilled into a simple dense student that is easy to deploy — even onto small language models that run on phones and laptops.

Isn’t that just fine-tuning?

Mechanically, response-based distillation is a fine-tuning run — what makes it distillation is where the training data comes from and what the goal is. Ordinary fine-tuning adapts a model to a task or domain using whatever data you have, usually human-curated. Distillation specifically uses a stronger model’s outputs as the curriculum, with the goal of transferring that model’s general capability into a smaller one. The mechanics overlap; the intent and the data source are what differ.

Distillation vs. quantization vs. pruning

Distillation is one of three standard compression levers, and they answer different questions:

	Distillation	Quantization	Pruning
What shrinks	Architecture (fewer/smaller layers)	Numeric precision of weights	Individual weights/structures removed
Requires training?	Yes — full training run for the student	No (or light calibration)	Usually needs fine-tuning to recover
Quality cost	Low if the student is sized sensibly	Low at 8-bit, grows as bits drop	Varies; aggressive pruning degrades
Best when	You want a genuinely smaller, faster model	You want the same model to fit smaller hardware	Sparsity-friendly hardware is available

They stack: a distilled student is routinely quantized afterward, compounding the savings. If you just need an existing model to fit on your GPU tonight, quantize. If you’re a lab building a cheap tier for millions of users, distill — then quantize that.

The limits

Distillation is not free capability. The known failure modes:

The student ceiling. A student can approach its teacher but not exceed it on the distilled behavior — and a small architecture may simply lack the capacity to absorb everything the teacher knows. Compression is lossy; the art is choosing which capabilities survive.

Distribution narrowing. A teacher’s outputs are a filtered, systematically-shaped slice of language. Train too many generations of models on model-written text and diversity erodes — errors and stylistic quirks get amplified rather than averaged out. Careful mixing with human-written data is standard practice for exactly this reason.

Compounding blind spots. Whatever the teacher gets wrong, the student learns as truth, without the surrounding knowledge that might let a bigger model recover.

The fine print. Most commercial providers’ terms of service restrict using their model outputs to train competing models. Distilling from your own models is routine; distilling from someone else’s API sits somewhere between license violation and open industry dispute, and it has been the subject of repeated public accusations between labs.

The takeaway

Model distillation transfers capability from a large teacher model into a small student by training the student on the teacher’s outputs — soft probability distributions in the classic formulation, synthetic worked examples in the LLM era. It’s how flagship-level behavior reaches the cheap, fast model tiers, and it composes with quantization for further savings. The constraint to remember: the student inherits the teacher’s ceiling and its blind spots. Distillation moves intelligence down-market; it doesn’t create more of it.