What Is HBM? High-Bandwidth Memory, Explained

High-Bandwidth Memory, or HBM, is a type of DRAM designed to deliver far more data per second than conventional memory. Instead of placing chips on a separate stick across the motherboard, HBM stacks multiple DRAM dies vertically — like a tiny skyscraper — and mounts the stack right beside the processor on the same package. The result is a memory system that can feed a chip with hundreds of gigabytes of data per second, roughly ten times what DDR5 can manage.

The memory wall

Modern AI accelerators are not compute-bound in the way people often assume. The arithmetic is fast; the bottleneck is feeding data to the cores fast enough. This problem is called the memory wall.

Training or running a large language model means continuously loading enormous weight matrices, multiplying them, and loading the next set. A GPU with thousands of tensor cores sits idle whenever it’s waiting for data to arrive from memory. Bandwidth — the rate at which data moves between memory and the chip — determines how much of that raw compute you can actually use. HBM exists to remove that bottleneck.

How stacking and TSVs work

Conventional DRAM places chips flat on a PCB and connects them with traces that run millimeters or centimeters to the processor. HBM takes a different approach:

1. Die stacking. Multiple DRAM dies (typically 8–12) are stacked vertically.
2. Through-silicon vias (TSVs). Tiny vertical copper pillars punch through each die, creating a direct electrical path from top to bottom of the stack — far shorter and wider than any PCB trace.
3. Silicon interposer. The memory stacks sit on a flat piece of silicon alongside the processor. The interposer’s fine-pitch wiring connects them at very high density, a technique called 2.5D packaging.

The physics are simple: shorter distance + wider bus = more bandwidth per watt. An HBM3E stack exposes a 1,024-bit-wide bus per stack; DDR5 uses a 64-bit bus per channel.

Generations

Generation	Max bandwidth (per stack)	Notes
HBM2E	~460 GB/s	Deployed in A100, MI200
HBM3	~819 GB/s	H100, MI300X
HBM3E	~1.2 TB/s	H200, MI325X; current standard
HBM4	~2+ TB/s	Sampling 2025, target for Blackwell Ultra and Rubin

HBM4 also widens the die-to-die interface further and moves to a base-die logic layer, which adds addressability and power management directly to the memory stack.

The trade-offs

HBM is not a universal upgrade — it comes with real costs:

• Price. A single HBM3E module costs roughly $60–100, versus $5–10 for a comparable amount of DDR5. The economics only make sense when bandwidth matters more than cost per gigabyte.
• Capacity. Stacking limits how many dies fit. A typical GPU ships with 80–192 GB of HBM today, where a server might carry terabytes of DDR5.
• Manufacturing complexity. TSVs and interposers require advanced packaging that only a handful of facilities can do. Yield challenges keep supply tight.

This is why HBM appears almost exclusively in AI accelerators, HPC chips, and high-end networking silicon — not in consumer laptops or servers doing general workloads.

A concentrated supply chain

Only three companies make DRAM: SK Hynix, Samsung, and Micron — and they collectively control roughly 95% of global supply. HBM is far more profitable per wafer, so all three have been reallocating capacity toward it. Around 23% of global DRAM wafer capacity now goes to HBM, a share that has roughly tripled since 2022.

The knock-on effect is tighter supply — and higher prices — for consumer DDR5 and server DRAM. This is part of what analysts call the AI memory supercycle: demand for HBM keeps growing faster than packaging capacity can scale. Micron’s recent supply agreement with Anthropic is one example of how chipmakers and AI labs are securing long-term allocations as a result.

The takeaway

HBM solves the memory wall by stacking DRAM dies vertically, connecting them with through-silicon vias, and placing the result inches from the processor on a silicon interposer. The bandwidth gains are dramatic — an order of magnitude over DDR5 — but so are the cost and manufacturing challenges. For AI accelerators, where the limiting factor is almost always how fast weights can be loaded, HBM is not optional. It is the reason modern GPU clusters can run models at the scale they do.