What Is a GPU? Why AI Runs on Graphics Chips

A graphics processing unit, or GPU, is a processor designed for parallel computation. Where a CPU has a handful of powerful cores optimized for running one instruction as fast as possible, a GPU has thousands of smaller cores — all doing arithmetic at the same time. That design was invented for rendering graphics, where every pixel in a frame needs the same geometric calculation performed independently. It turns out to be exactly what neural networks need too.

CPU vs. GPU: a design philosophy difference

The contrast comes down to latency versus throughput:

	CPU	GPU
Core count	8–128	Thousands to tens of thousands
Per-core speed	Very fast	Moderate
Optimized for	Sequential tasks, branching logic	Repeated parallel arithmetic
Memory model	Large system RAM	High-bandwidth on-package memory

A CPU is a generalist. It handles your operating system, web browser, database queries, and anything that involves unpredictable branching or low-latency responses. A GPU is a specialist: it runs the same operation on thousands of data points simultaneously, a programming model called SIMD (single instruction, multiple data).

Why neural networks map to GPUs so naturally

Training or running a neural network is, at its core, a lot of matrix multiplication. You have a matrix of weights, a matrix of input activations, and you multiply them together millions of times per forward pass. Matrix multiplication is embarrassingly parallel — each output element is independent of the others, so all of them can be computed at the same time.

Modern GPUs include dedicated tensor cores (NVIDIA’s term; AMD has equivalent hardware) that are purpose-built for these mixed-precision matrix operations. A single H100 GPU can perform roughly 3,958 teraflops of FP8 tensor operations per second. No CPU comes close.

The other half of the equation is memory bandwidth. Even with fast cores, a GPU stalls if it can’t load weight matrices quickly enough. That’s why modern AI chips use HBM — High-Bandwidth Memory stacked directly on-package — rather than the DDR5 used in servers and consumer desktops. The H200 ships with HBM3E delivering around 4.8 TB/s of bandwidth.

CUDA and NVIDIA’s software moat

Hardware alone doesn’t explain NVIDIA’s dominance. In 2006 the company released CUDA, a programming model and toolkit that let developers write general-purpose GPU code in C-like syntax. Over nearly two decades, CUDA became the foundation of the entire AI software stack: PyTorch, TensorFlow, JAX, and virtually every training framework target CUDA natively.

This creates a deep moat. Switching GPU vendors doesn’t just mean new hardware — it means porting the entire software pipeline. AMD’s ROCm and Intel’s oneAPI have made progress, but CUDA’s library ecosystem (cuDNN, NCCL, cuBLAS) remains the path of least resistance for most research teams.

Alternatives to GPUs

GPUs are not the only option for AI compute:

• TPUs. Google’s Tensor Processing Units are custom ASICs designed entirely around neural network workloads. They sacrifice generality for efficiency and are available through Google Cloud.
• NPUs. Neural Processing Units appear in consumer devices (Apple Neural Engine, Qualcomm Hexagon, AMD XDNA). They handle inference on-device — enough for voice transcription or image classification, not large-scale training.
• Custom inference chips. Companies like Cerebras, Groq, and SambaNova build silicon tailored to specific model architectures, trading flexibility for throughput at a fixed shape.

For the time being, GPUs — and NVIDIA GPUs in particular — remain the default for training large language models and running them at scale. The Rubin platform is NVIDIA’s next-generation architecture, targeting HBM4 and even higher compute density.

Beyond the data center

GPUs aren’t only for clouds and clusters. Tools like Ollama let developers run capable open-weight models on a consumer GPU. Meanwhile, WebGPU is bringing GPU-accelerated compute to the browser, enabling in-browser inference for smaller models without sending data to a server.

The takeaway

A GPU’s thousands of parallel cores make it the natural fit for the matrix arithmetic at the heart of neural networks. NVIDIA’s CUDA ecosystem amplified that hardware advantage into a software lock-in that the industry is still working around. Understanding GPUs — what they do, what limits them, and where alternatives compete — is essential context for anyone working with AI infrastructure, from cloud training runs down to on-device inference.