Model Distillation vs Quantization: Shrinking LLMs Compared

Model distillation and quantization are the two most common techniques for making a large language model cheaper to run, and they work in completely different ways. Distillation trains a new, smaller model to imitate the outputs of a larger “teacher” model — it changes the model’s architecture and parameter count. Quantization takes an existing model and represents its weights with fewer bits — it changes how precisely the same parameters are stored, without altering the model’s structure at all.

How distillation works

Distillation starts with a large, already-trained teacher model and a smaller student architecture — fewer layers, fewer parameters, or both. Instead of training the student only on hard labels (the “correct” next token), it’s trained to match the teacher’s full output distribution — the probabilities the teacher assigns across many possible next tokens, not just the single most likely one. This “soft target” signal carries more information than a single correct answer would: it tells the student not just what the right answer is, but how confident the teacher was and what the plausible alternatives were.

The result is a genuinely smaller model — fewer parameters, less memory, faster inference — that has learned to approximate the teacher’s behavior on the kinds of inputs it was trained on. It’s still a full training run, generally requiring a comparable data pipeline and compute budget to training any model of the student’s size, just guided by a stronger teacher signal instead of raw labels alone.

How quantization works

Quantization doesn’t touch the model’s architecture or parameter count at all. It takes the existing trained weights — typically stored as 16- or 32-bit floating-point numbers — and converts them to a lower-precision format, such as 8-bit or 4-bit integers. Each weight is mapped onto a smaller numeric range, which shrinks the model’s memory footprint and lets more of it fit in fast memory or run with cheaper arithmetic.

Because quantization only re-represents existing weights rather than retraining them, it’s dramatically cheaper to apply than distillation — often a post-processing step measured in minutes to hours rather than a full training run. The tradeoff is precision loss: representing a weight with fewer bits introduces rounding error, which can degrade output quality if pushed too far. Well-designed quantization schemes calibrate which layers can tolerate lower precision and which need to stay closer to their original format, which is why quality loss from moderate quantization is often small relative to the size and speed gains.

Comparing the two approaches

	Distillation	Quantization
What changes	Model architecture and parameter count	Numeric precision of existing weights
Requires retraining	Yes — a full training run against a teacher	No — typically a post-processing step
Cost to apply	High (comparable to training a model from scratch)	Low (fast, often automated)
Memory savings	Large (fewer parameters)	Large (same parameters, smaller storage)
Typical quality tradeoff	Depends heavily on student size vs teacher	Small at moderate precision, larger if pushed too far
Can combine with the other	Yes	Yes

Why they’re often used together

Distillation and quantization solve different problems, which is why production deployments frequently apply both: distill a large teacher into a smaller student model, then quantize that student further for deployment. The distilled model is architecturally smaller to begin with; quantizing it on top compounds the memory and speed savings without requiring a second full training run. This combination shows up often in small language models designed to run on constrained hardware, including setups for running LLMs locally on a single machine.

Where each one fits

Quantization is usually the first lever to reach for, because it’s cheap and reversible in the sense that you can always keep the original full-precision weights around. It’s the right tool when you already have a model that performs the way you want and just need it to run faster or fit in less memory — for instance, moving a model from a data-center GPU to a more constrained inference environment.

Distillation is the right tool when you need a model that’s structurally smaller — for latency reasons, cost-per-token reasons, or because you’re deploying to hardware where even a quantized version of the full model won’t fit — and you’re willing to pay the upfront cost of a dedicated training run. It’s also useful when you want a smaller model specialized for a narrower task than the teacher was built for, since the student only needs to learn the teacher’s behavior on the distribution of inputs it’s trained against, not the teacher’s full general-purpose capability.

What neither technique is

Neither distillation nor quantization is the same as fine-tuning with LoRA, which adapts an existing model to a new task or domain without necessarily shrinking it. It’s possible to combine all three: distill a smaller student, quantize it for deployment, and then LoRA fine-tune it for a specific use case — each technique addresses a different axis (capability transfer, numeric precision, task adaptation) and they compose rather than compete.

The takeaway

Distillation trains a new, smaller model to imitate a larger one’s behavior, requiring a real training run but yielding genuine architectural savings. Quantization keeps the same model and the same architecture but stores its weights with fewer bits, which is far cheaper to apply but has a lower ceiling on how much it can shrink a model before quality degrades noticeably. Most production LLM deployments use both — a distilled model that’s quantized on top of that — because the two techniques target different sources of cost and stack cleanly.