How Backpropagation Works in Neural Networks

Backpropagation is the algorithm neural networks use to figure out how much each individual weight contributed to a prediction error, so that gradient descent knows which direction to adjust every weight. It’s not a training algorithm on its own — it’s the piece that makes gradient descent computationally feasible for networks with millions or billions of parameters, by efficiently computing the gradient of the loss with respect to every weight in a single backward pass through the network.

The problem backpropagation solves

Training a neural network means adjusting its weights to reduce the difference between its predictions and the correct answers — a quantity called the loss. Gradient descent tells you the general recipe: compute the gradient (the slope) of the loss with respect to each weight, then nudge each weight a small step in the direction that reduces the loss. The catch is computing that gradient. A modern network can have millions of weights spread across dozens of layers, and the effect of any single early-layer weight on the final loss is buried under every layer of computation that happens after it.

Computing each weight’s gradient independently, from scratch, would be prohibitively expensive — it would mean re-running the entire network once per weight just to see how a tiny nudge to that one weight changed the output. Backpropagation avoids that by computing all the gradients in a network in one pass, reusing intermediate results as it goes.

The chain rule, applied layer by layer

Backpropagation is really just a systematic application of the calculus chain rule to a layered computation. A neural network’s output is a composition of functions — each layer takes the previous layer’s output and applies a weighted transformation and an activation function. The chain rule says that to find how a change in an early layer’s weight affects the final loss, you multiply together the local derivatives at each layer between that weight and the output.

The algorithm has two phases:

1. Forward pass — input data flows through the network layer by layer, producing a prediction. The loss function compares that prediction to the correct answer and produces a single error value.
2. Backward pass — starting from that error, the algorithm works backward through the network, layer by layer, computing how much each layer’s inputs contributed to the error using the chain rule. Each layer’s local gradient is computed once and reused for every weight in that layer, and passed backward to the layer before it.

By the time the backward pass reaches the first layer, every weight in the network has an associated gradient — the direction and magnitude by which changing that weight would move the loss. Gradient descent then uses those gradients to update every weight simultaneously.

Why this is efficient

The efficiency comes from reusing computation rather than repeating it. Each layer’s contribution to the gradient is computed exactly once, and that result is combined with gradients flowing back from the next layer rather than recomputed. This means the cost of computing gradients for an entire network scales with roughly the same cost as a single forward pass, rather than growing with the number of weights — a difference that makes training networks with billions of parameters practical instead of computationally impossible.

This is also why frameworks like PyTorch and TensorFlow build a computation graph during the forward pass: the graph records exactly which operations were applied, so the backward pass can walk it in reverse and apply the chain rule automatically, without a developer hand-deriving gradients for every architecture.

Backpropagation and the vanishing gradient problem

One well-known failure mode of backpropagation in deep networks is the vanishing gradient problem: because gradients are computed as a product of many local derivatives, each one typically a fraction less than 1, the product can shrink toward zero as it propagates back through many layers. Early layers in a very deep network can end up with gradients too small to meaningfully update their weights, effectively freezing them. This was a major obstacle to training deep networks until architectural fixes — residual connections, normalization layers, and better activation functions — became standard. It’s also part of the motivation behind self-attention, which gives transformer layers more direct paths for gradients to flow, in contrast to older recurrent architectures where gradients had to travel through many sequential steps.

Backpropagation’s relationship to other training concepts

Backpropagation computes gradients; it doesn’t decide how big a step to take with them, or when to stop training. That’s the job of the optimizer (which applies gradient descent, often with refinements like momentum or adaptive learning rates) and of training practices aimed at avoiding overfitting, such as regularization and early stopping. It’s also distinct from techniques applied after training, like neural network pruning, which removes weights once training is done rather than computing how to adjust them. And it’s the same underlying mechanism whether you’re training a small classifier or a large transformer or mixture-of-experts model — the chain rule doesn’t care how large or exotic the architecture is, only that every operation in the forward pass has a defined derivative.

The takeaway

Backpropagation is the mechanism that makes training large neural networks computationally tractable: it applies the chain rule once, backward through the network, to compute how every weight contributed to the final error, reusing intermediate results along the way instead of recomputing each weight’s effect from scratch. Gradient descent then uses those gradients to update the weights. Understanding backpropagation is really understanding why deep learning became practical at scale — the algorithm turns an otherwise combinatorially expensive problem into one that costs about as much as a single forward pass.