Backpropagation

Overview

Backpropagation is the learning mechanism at the heart of deep learning. A network makes a prediction, the prediction is compared against the correct answer, and backpropagation is how the resulting error is turned into a precise instruction for improving every one of the network's weights. Without it, a model with billions of parameters would have no principled way to discover which of those parameters to change, or by how much.

How it works

The process runs in two passes. In the forward pass, data flows through the layers and produces an output, from which a loss is calculated. In the backward pass, the algorithm applies the chain rule of calculus to work out, layer by layer from the output toward the input, how much each weight contributed to that loss. Those contributions, the gradients, tell an optimizer exactly which direction to nudge each weight to reduce the error next time.

Why it matters

Backpropagation is computationally relentless, every weight must be updated on every training step, across millions or billions of examples. This is precisely the dense, parallel arithmetic GPUs were built to perform, and it is why training time is governed as much by memory bandwidth and interconnect speed as by raw core count. The faster the gradients can be computed and exchanged between GPUs, the faster a model converges.