Derivatives and gradients
A derivative says how much the output changes when you nudge an input. A gradient is the list of derivatives for all inputs. Training uses it to know which way to adjust each weight.
Words on this page (8)
- gradient
- : The direction and rate at which the loss changes when each parameter changes. Training steps against it.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- backpropagation
- : The method that computes the gradient for every parameter by applying the chain rule backward through the network.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
- ReLU
- : An activation function that outputs the input if positive, otherwise 0.
- neuron
- : One unit in a network: it multiplies inputs by weights, adds a bias and applies an activation function.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
The idea
For f(x) = x², the derivative is 2x. At x = 3 the slope is 6: increase x by a tiny 0.01 and f goes up by about 0.06. A function with several inputs, like f(a, b) = a² + 3b, has a partial derivative for each: 2a with respect to a, and 3 with respect to b. Stack them and you get the [2a, 3]. It points in the direction of steepest increase. To reduce the function, step the opposite way.
Common rules
Power: the derivative of xⁿ is n·xⁿ⁻¹. Sum: derivatives add. Constant multiple: the constant stays. Chain: the derivative of f(g(x)) is f′(g(x))·g′(x). Those four are enough to differentiate every operation in a network by hand.
Example
f(a, b) = a² + 3b at (a, b) = (2, 1) gives f = 7. Gradient = [4, 3]. Moving a by +0.1 raises f by about 0.4. Moving b by +0.1 raises it by 0.3. To lower f, move a and b in the negative direction, with a bigger step on a.
Check numerically
You can estimate a derivative as (f(x + h) - f(x)) / h for a tiny h like 0.0001. This numerical check is how people test that a hand-derived gradient is right.
Worked example: a tiny slope
f(x) = x². At x = 3, f is 9. Nudge x to 3.01 and f becomes 9.0601. The change is 0.0601 for a nudge of 0.01, a ratio of about 6. The derivative 2x gives 6 exactly. The slope tells you that increasing x makes f grow quickly there, so to reduce f you should decrease x.
Why many directions
A model has billions of . The gradient has one slope per weight: for each, "if I nudge this weight up, does the error go up or down, and how fast?" One pass computes all of them at once, which is the trick behind .
How the chain rule traces through
Take f(x) = (3x + 1)² at x = 2. Think of two steps: g = 3x + 1, then f = g². At x = 2, g = 7 and f = 49. The outer slope is 2g = 14. The inner slope is 3. Multiply: 14 × 3 = 42. Numerical check: at x = 2.001, g = 7.003, f = 49.042, so the change is 0.042 for a nudge of 0.001, a ratio of about 42. This multiply-the-slopes habit is all that backpropagation does.
A second example: where the slope is zero
At x = 0, f(x) = x² has slope 0, and that is the bottom of the bowl, so no step is needed. But zero slope is not always a good sign. f(x) = x³ also has slope 0 at x = 0, yet it is not a bottom. Values to the left are lower and values to the right are higher, so the slope is zero only for a moment and the curve keeps climbing. In a network, a with a negative input has slope 0, so no gradient flows through it and its weights stop learning. Flat does not always mean finished.
What goes wrong: slopes that multiply
By the chain rule, the slope through a deep stack is a product of one slope per layer. Invented numbers: if each of 10 layers contributes 0.25, the product is 0.25¹⁰ = 0.00000095. The early layers barely get a signal, called a vanishing gradient. If each contributes 2, the product is 2¹⁰ = 1,024, an exploding gradient. Symptoms: that does not move, or loss that suddenly becomes nan. Confirm by printing the gradient size for each layer and looking for a steady drop or rise toward the start of the network.
Check it by hand
Verify the gradient of f(a, b) = a² + 3b at (2, 1) the way a code test would, with h = 0.001. Raise a: f(2.001, 1) = 4.004001 + 3 = 7.004001, so the slope is 0.004001 / 0.001 = 4.001, close to 4. Raise b: f(2, 1.001) = 7.003, so the slope is 3. Both match [4, 3]. A very tiny h such as 10⁻¹² can do worse, because rounding errors in the computer swamp the change.
Common mistake
Mixing up direction. The gradient points uphill. To reduce error you move against it.
You are standing on a foggy hill and want to get down. You cannot see the valley, but you can feel which way the ground slopes under your feet. You take a step downhill, feel again, and repeat. The slope under your feet is the derivative. With many directions at once, the list of slopes is the gradient. Training an AI model is exactly this walk.
Check yourself
What is the derivative of f(x) = x² at x = 3?
Not quite. The derivative is 2x, so at x = 3 it is 6.
In which direction does the gradient point?
Not quite. You step against it to reduce a function, which is what training does.
f(a, b) = a² + 3b at (2, 1). Which nudge raises f the most per unit step?
Not quite. The gradient is [2a, 3] = [4, 3], so a changes f faster here.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · The intuition for what a derivative is.
- Video
Eugene Khutoryansky · Animated partial derivatives and the gradient vector.
- CourseEssence of calculusCourse
3Blue1Brown · The full visual series.
Spotted a mistake or something unclear? Report it on GitHub.