What Is Quantization? Smaller, Faster AI Models

Quantization is the process of reducing the numeric precision of a neural network’s weights — and sometimes its intermediate values — to use less memory and run faster. A typical large model stores each weight as a 16-bit or 32-bit floating-point number; quantization replaces those with 8-bit integers, 4-bit integers, or even lower, shrinking the model to a fraction of its original size. It is the key technique that makes running capable LLMs on consumer hardware practical.

Why precision can be reduced

Neural network weights are not uniformly distributed across the full range of floating-point values. They cluster, they repeat, and small rounding errors in individual weights average out across billions of parameters. That redundancy means you can round values to a coarser grid — quantize them — and the model still produces nearly the same outputs, because the collective behavior of billions of slightly-rounded weights remains close to the original.

The trade-off is real but often acceptable: 4-bit quantization typically costs 1–3 points on standard benchmarks, and for most practical tasks, users cannot tell the difference.

The two main approaches

Post-training quantization (PTQ) takes an already-trained model and maps its weights into lower-precision formats without retraining. It is fast and cheap — you just load the weights and compress them. Most of the quantized models you download today are PTQ. The cost is a modest accuracy drop, which grows larger at very aggressive bit widths (2-bit and below).

Quantization-aware training (QAT) simulates quantization noise during training itself, so the model learns to be robust to the rounding. QAT produces higher-quality quantized models than PTQ at the same bit width, but it requires access to training infrastructure and data — something most end users and even most inference providers don’t have.

Common formats

The ecosystem has settled on a few standard formats for distributing quantized LLMs:

Format	Description
GGUF	The standard format for running quantized models locally with llama.cpp and tools like Ollama. Supports a range of bit widths (Q4_K_M, Q5_K_M, Q8_0, etc.) in a single portable file.
GPTQ	GPU-optimized PTQ; produces high-quality 4-bit models suited for inference with Transformers or vLLM on CUDA hardware.
AWQ	Activation-Aware Weight Quantization; a PTQ method that identifies which weights matter most and preserves more precision for them, reducing accuracy loss at 4-bit.

The Q4_K_M naming convention in GGUF, for example, means 4-bit weights with a “medium” K-quant strategy — a balance of quality and size that has become one of the most popular choices for local inference.

8-bit vs. 4-bit

8-bit (INT8) quantization roughly halves a model’s memory footprint compared to FP16. Accuracy loss is typically under 1% on most tasks. It is the conservative choice when hardware allows it.

4-bit (INT4) halves it again — a 70B-parameter model that needs ~140 GB in FP16 fits in roughly 35 GB at 4-bit. This is what makes a 70B model runnable on a single high-end consumer GPU or a Mac Studio. The accuracy cost is larger but still acceptable for most conversational and coding tasks.

Below 4-bit, quality degrades faster than size decreases, so 2-bit and 1-bit remain research territory.

What it enables in practice

Running LLMs locally on a laptop or desktop was impractical before 4-bit quantization. A 7B model at FP16 needs ~14 GB of GPU VRAM, but at Q4 it drops to ~4 GB — something a gaming GPU can handle. The same dynamic applies on the server side: quantization lets providers fit more model capacity on the same HBM-constrained accelerator, lowering per-token cost.

Quantization is distinct from KV-cache compression (which reduces the memory cost of the attention cache at inference time, not the weights), though both attack the same hardware bottleneck. Research like Google’s TurboQuant is pushing both techniques forward simultaneously to squeeze more out of each chip.

Vector embeddings used in semantic search and RAG can also be quantized — storing embedding vectors as INT8 instead of FP32 saves 75% of the index memory with minimal retrieval quality loss, applying the same core idea to a different part of the AI stack.

The takeaway

Quantization trades a small amount of model quality for large gains in memory efficiency and inference speed. At 4-bit, models run on hardware that would otherwise be out of reach, and at 8-bit the cost is nearly invisible. Understanding what format and bit width a model uses tells you a lot about where it will run, how fast, and what accuracy to expect — and that knowledge matters whether you are choosing a local model or deploying one at scale.