Entropy and KL divergence
Entropy measures surprise. Cross-entropy is the loss language models train on. KL divergence measures how far one distribution is from another.
Words on this page (8)
- cross-entropy
- : The standard loss for next-token prediction: minus the log of the probability given to the correct token.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- nats
- : The unit of information when you use the natural logarithm. Bits use log base 2.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- RLHF
- : Reinforcement learning from human feedback: tune a model toward answers that people rate higher.
- batch
- : A group of examples processed together in one step.
The ideas
A answers the question: what power must I raise the base to, to get this number? Since 2³ = 8, log₂(8) = 3. Log base 2 gives bits, and the natural log (base e) gives . code usually uses the natural log, so the lesson's 0.9 gives 0.105 nats, which is 0.152 bits. Same quantity, different unit. Now: an event with probability p carries surprise of -log₂(p) bits. A fair coin flip has surprise 1 bit. An event with probability 1/8 has 3 bits. Entropy is the average surprise of a distribution: H(p) = -Σ p·log₂ p. A fair coin has entropy 1 bit, a coin that always lands heads has 0.
H(p, q) = -Σ p·log q is the average surprise when the truth is p but you use model q. For next-token prediction the truth puts probability 1 on the actual , so cross-entropy reduces to minus the log of the probability the model gave it. That is the training loss.
KL divergence is the gap: KL(p ‖ q) = H(p, q) - H(p). It is 0 only when q equals p, and always at least 0. It shows up when you want to keep a fine-tuned model close to its base, as in the penalty used in .
Example
True distribution p = [0.5, 0.5]. Model q = [0.9, 0.1]. H(p) = 1 bit. H(p, q) = -(0.5·log₂0.9 + 0.5·log₂0.1) = 0.5×0.152 + 0.5×3.322 = 1.737 bits. KL = 1.737 - 1 = 0.737 bits. Perplexity, a common language model score, is 2 raised to the cross-entropy, so lower is better.
In plain words
Cross-entropy asks: if reality follows distribution p but I use my model q to predict it, how surprised will I be on average? The better q matches p, the smaller the number. KL divergence is just the extra surprise caused by using q instead of the truth.
Worked example in model terms
The real next word is "Paris". Model A gave "Paris" 90 percent: surprise of 0.15 bits (0.105 nats). Model B gave it 5 percent: surprise of 4.3 bits. Averaged over millions of sentences, that gap is the difference between a good model and a bad one.
Surprise by example
Surprise is -log₂(p) bits. Probability 0.5 gives 1 bit, 0.25 gives 2, 0.125 gives 3, 0.9 gives 0.152, 0.1 gives 3.322 and 0.05 gives 4.322. Rare events carry more surprise, and certain events carry none. Entropy averages it. A coin with probabilities [0.9, 0.1] has entropy 0.9 × 0.152 + 0.1 × 3.322 = 0.137 + 0.332 = 0.469 bits, far under the fair coin's 1 bit, because you can mostly guess the result. A coin with [0.25, 0.75] has 0.25 × 2 + 0.75 × 0.415 = 0.811 bits.
A second example: the direction matters
Use p = [0.5, 0.5] and q = [0.9, 0.1]. KL(p ‖ q) = 0.5 × log₂(0.5/0.9) + 0.5 × log₂(0.5/0.1) = 0.5 × (-0.848) + 0.5 × 2.322 = 0.737 bits. Swap them: KL(q ‖ p) = 0.9 × log₂(0.9/0.5) + 0.1 × log₂(0.1/0.5) = 0.9 × 0.848 + 0.1 × (-2.322) = 0.531 bits. Two different numbers. The first asks what it costs to use q when reality is p, the second the reverse. In RLHF, which direction is penalized changes the behavior.
What perplexity means
Perplexity is 2 raised to the entropy, and reads as "how many equally likely choices the model is torn between". A model choosing among 4 equal options has entropy 2 bits and perplexity 2² = 4. A model that always puts all its probability on the right token has perplexity 1. Numbers from different tokenizers are not comparable, because the model predicts different pieces.
What goes wrong
If reality allows an outcome but the model gives it probability 0, the surprise is infinite, and so are cross-entropy and KL. A training log showing inf at one often points to this. The symptom is a single huge spike rather than a slow rise. Another trap is mixing units: 0.105 nats and 0.152 bits are the same quantity, so compare numbers only in one unit.
Common mistake
Thinking KL divergence is a distance like on a map. It is not symmetric: KL(p, q) and KL(q, p) are different numbers.
You have a coin. If it always lands heads, the next flip tells you nothing, so there is no surprise. If it is fair, each flip is a surprise worth one bit. Entropy is how surprised you expect to be on average. A language model that is good at predicting the next word is rarely surprised, so its loss is low.
Check yourself
How much entropy does a fair coin have?
Not quite. Each flip carries one bit of surprise on average.
What is the KL divergence between two identical distributions?
Not quite. There is no gap when q equals p.
A model gives the real next word 90 percent in one run and 5 percent in another. Which run has the larger loss?
Not quite. Loss is minus the log of the probability given to the right word, so a small probability means a big loss.
You got 0 of 3 right.
Go deeper
- Video
Aurélien Géron · Ten minutes, with the coin and weather examples.
- ArticleVisual Information TheoryArticle
Chris Olah · Draws entropy and cross-entropy as areas.
- ArticleKullback-Leibler divergenceArticle
Wikipedia · Definition, properties and why it is not symmetric.
Spotted a mistake or something unclear? Report it on GitHub.