Backpropagation
Backprop is the chain rule applied efficiently, so one backward pass gives the gradient for every weight.
Words on this page (10)
- backpropagation
- : The method that computes the gradient for every parameter by applying the chain rule backward through the network.
- gradient
- : The direction and rate at which the loss changes when each parameter changes. Training steps against it.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- parameter
- : One learned number inside a model. Modern models have billions of them. Training adjusts them.
- inference
- : Running a trained model to produce output, as opposed to training it.
- matrix
- : A grid of numbers. Multiplying a vector by a matrix transforms it. Model layers are mostly matrices.
- gradient descent
- : Repeatedly nudging parameters opposite to the gradient to reduce the loss.
- Adam
- : The most common optimizer. It smooths gradients and adapts the step size for each parameter.
The idea
A network is a chain of simple operations. The chain rule says the derivative of a composition is the product of the derivatives of its parts. For y = f(g(x)), dy/dx = f′(g(x)) · g′(x). runs the network forward to get the , then walks backward through the operations, multiplying local derivatives and reusing each partial result.
Example
Let a = w · x, then p = a + b, and L = (p - t)². With x = 2, w = 1, b = 0, t = 5: a = 2, p = 2, L = 9. Going backward: dL/dp = 2(p - t) = -6. dp/da = 1, so dL/da = -6. da/dw = x = 2, so dL/dw = -12. Update with η = 0.1: w = 1 + 1.2 = 2.2. Now a = 4.4 and L = 0.36. One step cut the loss from 9 to 0.36.
Why it matters
The cost of the backward pass is about the same as the forward pass, no matter how many there are. That is what makes billions of feasible. Frameworks like PyTorch compute it automatically, so you rarely write it, but knowing what it does explains memory use and why training is far more expensive than .
Why it is fast
Computing each weight's blame separately would take billions of passes. Backprop reuses work: the blame at a later layer is passed backward and combined with each earlier layer's local slope. One forward pass and one backward pass give every . That is why training a huge model is possible at all.
A second step of the same example
Continue from the example: after the first update w = 2.2. Forward: a = 2.2 × 2 = 4.4, p = 4.4, L = (4.4 - 5)² = 0.36. Backward: dL/dp = 2(4.4 - 5) = -1.2, and dL/dw = -1.2 × 2 = -2.4. Update: w = 2.2 + 0.1 × 2.4 = 2.44. Forward again: a = 4.88 and L = 0.0144. The same backward pass also gives the bias its gradient: dL/db = dL/dp × dp/db, and dp/db = 1, so it is -6 on the first step. The numbers above update only w and keep b = 0. If you updated b as well (b = 0.6), the next prediction would be 5.0 and the loss 0. One backward walk serves every parameter.
When one weight is used twice
If a weight appears in two places, its gradient is the sum of the gradients along each path. Take p = w×x1 + w×x2 with x1 = 2, x2 = 3, target t = 10 and w = 1. Then p = 5 and L = 25. dL/dp = 2(5 - 10) = -10. The path through x1 contributes -10 × 2 = -20, and the path through x2 contributes -10 × 3 = -30. So dL/dw = -50. With η = 0.01, w = 1.5, p = 7.5 and L = 6.25. This summing is how one filter reused across a whole image, or one weight reused across a whole sentence, gets a single gradient.
Trade-offs: memory and cost
The backward pass needs numbers from the forward pass. In the example, da/dw = x, so x must be kept. A real network keeps these stored values for every layer until the backward pass reaches them. That is a large part of why training needs much more memory than running the model. A common trick is to throw some away and recompute them later, trading extra compute for less memory.
How to tell it is working
Test your gradient numerically. For L(w) = (2w - 5)², nudge w from 1 to 1.001. L goes from 9 to 8.988004, a change of -0.011996, and dividing by 0.001 gives -11.996. Backprop said -12. If a hand-written backward step disagrees by more than a rounding error, it has a bug. In PyTorch, one more thing to know: gradients add up across backward calls unless you reset them each step, a common cause of training that looks wrong.
Common mistakes
Believing backprop is a learning algorithm by itself. It only computes the gradients. (or ) uses them to change the weights.
A restaurant gets a bad review: the soup was cold. Who is to blame? The chef, the waiter who waited, the oven? You trace backward through the chain to see how much each step contributed. Backpropagation does this for a network: when the answer is wrong, it walks backward through the layers and tells every weight how much it contributed to the mistake.
Check yourself
dL/da = -6 and da/dw = 2. What is dL/dw?
Not quite. By the chain rule you multiply the local derivatives along the path.
Why is backpropagation efficient?
Not quite. The cost is about one extra pass, however many weights there are.
After one update, the loss goes from 9 to 0.36. What does that tell you about the step?
Not quite. A large drop in loss from one step means the direction and size were good.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · The intuition for how gradients flow backward.
- Video
3Blue1Brown · The chain rule with actual derivatives.
- Repo
Andrej Karpathy · The smallest working implementation to read.
Spotted a mistake or something unclear? Report it on GitHub.