Articles

What Is CXL (Compute Express Link)?

CXL is an interconnect standard that lets CPUs, GPUs, and memory devices share coherent memory over PCIe, enabling memory pooling and expansion.

Chisato Chisato · · 4 min read
A close-up of a computer memory module

CXL, or Compute Express Link, is an open interconnect standard built on top of the physical PCIe link that lets CPUs, GPUs, accelerators, and memory devices share memory coherently — meaning multiple devices can read and write the same memory region with a consistent, up-to-date view of it, something plain PCIe was never designed to guarantee. CXL is what makes it practical to attach large pools of extra memory to a server, or share memory directly between a CPU and an accelerator, without routing everything through slow, explicit data copies.

Why PCIe alone wasn’t enough

PCIe is extremely good at what it was designed for: moving data between a CPU and a peripheral device, like a network card, storage controller, or GPU, over a fast serial link. But PCIe was built around the idea of independent address spaces — a device’s memory and the host CPU’s memory are separate, and getting data from one to the other means an explicit copy or DMA transfer.

That model works fine for a network card streaming packets, but it breaks down for use cases where a CPU and an accelerator need to work on the same data structure simultaneously, or where you want to treat memory attached to a completely separate device as if it were ordinary system RAM. Cache coherency — keeping every device’s view of shared memory consistent as it’s written from multiple places — is the piece PCIe never provided. CXL adds it on top of the same physical PCIe layer, so it reuses existing connectors and signaling while adding a new protocol layer for coherency.

The three protocols under the CXL umbrella

CXL actually bundles three sub-protocols that run over the same physical link, each suited to a different kind of traffic:

  • CXL.io — essentially PCIe as usual: device discovery, configuration, and traditional non-coherent data transfer. This is the compatibility baseline.
  • CXL.cache — lets an attached device (like an accelerator) cache the host CPU’s memory coherently, so the device can read host memory efficiently and the CPU always sees the correct, up-to-date value even if the device has a cached copy.
  • CXL.mem — lets the host CPU access memory that physically lives on an attached device as if it were ordinary system memory, with full coherency. This is the protocol behind memory expansion and pooling.

Different classes of CXL devices use different combinations of these three: a memory-expansion module mostly relies on CXL.mem, while a coherent accelerator uses CXL.cache alongside it.

What this actually enables

Memory expansion. A server’s memory capacity is normally capped by how many DIMM slots its motherboard has. CXL lets a server attach additional memory modules over a CXL link instead, adding capacity beyond what’s physically socketed to the CPU — useful for memory-hungry workloads like large in-memory databases or vector databases that would otherwise be capacity-constrained.

Memory pooling. Multiple servers in a rack can share access to a common pool of CXL-attached memory, rather than each server owning a fixed, often underutilized allocation of its own RAM. A workload that temporarily needs more memory than its own server has installed can draw from the shared pool instead of that memory sitting idle in a neighboring, lightly-loaded server. This is a similar idea in spirit to how a load balancer spreads compute demand across servers, applied to memory capacity instead.

Coherent accelerator memory. GPUs and other accelerators traditionally have their own dedicated memory, separate from system RAM, with data explicitly copied back and forth across PCIe. CXL.cache and CXL.mem allow an accelerator and the host CPU to share a coherent view of memory instead, cutting out some of that copying overhead for workloads that pass data back and forth frequently between CPU and accelerator logic.

Plain PCIeCXLProprietary accelerator interconnects (e.g. NVLink)
Cache coherencyNoYesYes
Physical layerPCIePCIe (adds a protocol layer)Proprietary
Vendor opennessOpen standardOpen standard, multi-vendorVendor-specific
Primary use caseGeneral device I/OMemory expansion, pooling, coherent accelerator accessHigh-bandwidth GPU-to-GPU links within a vendor’s ecosystem
Typical adoptersNearly universalServer CPU and memory vendors converging on itLocked to a specific accelerator vendor’s hardware

CXL’s pitch is specifically that it’s an open, multi-vendor standard riding on ubiquitous PCIe hardware, rather than a proprietary link tied to one company’s silicon — the tradeoff being that a shared industry standard typically moves more slowly and conservatively than a single vendor iterating on its own closed interconnect.

Where this matters in practice

Memory has become an increasingly binding constraint in data centers, particularly for workloads shaped by the same forces driving HBM demand in AI accelerators — capacity and bandwidth both matter, and DIMM slots and accelerator memory alike are physically limited. CXL memory expansion and pooling are one of the tools cloud providers and server vendors are converging on to decouple memory capacity from the fixed number of slots on any one motherboard, similar in motivation to how NUMA architectures already manage memory that’s physically distant from a given CPU core, just extended to memory living on an entirely separate device over an external link.

The takeaway

CXL layers cache coherency on top of the same physical PCIe connectors already in every server, splitting into CXL.io, CXL.cache, and CXL.mem sub-protocols that together let CPUs, accelerators, and external memory devices share a consistent view of memory. The practical payoff is memory that can be expanded, pooled, and shared with coherent accelerator access beyond what’s physically socketed into a single server — an open, multi-vendor answer to a memory-capacity problem that’s only getting more pressing as workloads grow.

Chisato Chisato · · 5 min read

Cache Coherence and the MESI Protocol, Explained

Cache coherence keeps each CPU core's private cache consistent with the others. The MESI protocol is the classic mechanism that makes it work.

#Hardware #Computer Science #Chips