Loss and gradient descent

The ideaCopy Anchor Link

Loss is a number that says how far the output is from the target. For one example with prediction p and target t, squared error is L = (p - t)². Training adjusts the weights w to make L small. The gradient dL/dw says how L changes if w increases. Gradient descent moves against it: w ← w - η · dL/dw, where η is the learning rate.

For language models the loss is cross-entropy: minus the natural logarithm (ln) of the probability the model gave the correct next token. If it gave the right token probability 0.9, loss is 0.105. The unit is called nats. The next lessons use bits instead, which is the same idea with a different logarithm. At probability 0.1, loss is 2.3.

ExampleCopy Anchor Link

Minimize L(w) = (w - 3)². The gradient is 2(w - 3). Start at w = 0 with η = 0.1: gradient is -6, so w becomes 0 + 0.6 = 0.6. Next, gradient -4.8, w = 1.08. Then 1.46, 1.77, and so on toward 3. With η = 1.1 it overshoots and diverges. With η = 0.001 it barely moves. The learning rate is a core hyperparameter.

At scaleCopy Anchor Link

The same loop runs on billions of weights using mini-batches of data. Optimizers like Adam adapt each step. Backpropagation is how the gradients for all layers are computed efficiently.

Try itCopy Anchor Link

Use the demo: pick a learning rate and click step. Try a rate that is too large.

What the loss number meansCopy Anchor Link

Loss is the score of how wrong the model is. Zero would be perfect. If the model gave the right next word a probability of 0.9, the loss is small (0.105). If it gave 0.01, the loss is large (4.6). Training pushes the loss down, which means the model gives more probability to what really came next.

Choosing the learning rateCopy Anchor Link

Too small and training crawls, like walking down a mountain an inch at a time. Too large and you leap over the valley and land higher up on the other side. Practitioners usually start with a small value, watch the loss curve, and use a schedule that begins gently and shrinks over time.

Why each step shrinks the errorCopy Anchor Link

In the example, the gradient is 2(w - 3). One update gives w_new - 3 = (w - 3) × (1 - 2η). So with η = 0.1 the distance to the goal is multiplied by 0.8 each step: 3, 2.4, 1.92, 1.536. The values of w are 0, 0.6, 1.08, 1.464, 1.771, 2.017. The shrink factor tells you the learning rate story. With η = 0.25 the factor is 0.5, and w goes 0, 1.5, 2.25, 2.625. With η = 0.5 the factor is 0, so w lands on 3 in one step. With η = 1 the factor is -1, and w bounces between 0 and 6 forever. With η = 1.1 the factor is -1.2, so the gap grows each time.

A second example: noisy mini-batchesCopy Anchor Link

Real training does not use the whole dataset per step. It averages the gradient over a mini-batch, a small sample. Invented gradients for one weight on four examples: -6, -2, -4, -4. The average is -16 / 4 = -4. Another batch may give -3.5. The steps are slightly off from the true direction but cheap, and over many steps they average out. A bigger batch gives a steadier gradient but costs more per step.

What goes wrongCopy Anchor Link

Three failures, each with a symptom.

1. Loss becomes nan or shoots upward: the learning rate is too big. Confirm by halving it and rerunning.
2. Loss barely moves: the rate is too small, or the gradients are zero because of a bug. Confirm by checking whether weights change after a step.
3. Training loss falls while held-out loss rises: memorizing. Confirm by plotting both curves. Falling loss only proves the model fits its training numbers, not that it is useful.

Check it by handCopy Anchor Link

Run the lesson's example with η = 0.25 on paper, from w = 0. The gradient 2(w - 3) is -6, so w = 0 + 1.5 = 1.5. Next the gradient is -3, so w = 2.25. Next -1.5, so w = 2.625. The distance to 3 halves each step: 3, 1.5, 0.75, 0.375.

Common mistakeCopy Anchor Link

Changing the learning rate and the data in the same run. When the loss improves you cannot tell which change helped, so alter one thing per run.