Probability and logarithms, a quick refresher

Probability in one minuteCopy Anchor Link

A probability is a number from 0 (never) to 1 (certain). The probabilities of all the possible outcomes add up to 1. If two things are independent, the chance that both happen is the product. A fair coin gives heads with probability 0.5, so two heads in a row is 0.5 × 0.5 = 0.25. A language model gives every possible next token a probability, and those add up to 1. The probability of a whole sentence is the product of the probabilities of its tokens.

Logarithms in one minuteCopy Anchor Link

A logarithm answers the question: what power do I raise the base to, to get this number? Since 2³ = 8, log base 2 of 8 is 3. Since 10² = 100, log base 10 of 100 is 2. The natural logarithm, written ln, uses the special number e, about 2.718, as its base. Two facts matter here. The log of a product is the sum of the logs: log(a × b) = log a + log b. And the log of a number between 0 and 1 is negative, more so the smaller the number.

A worked example: why models use logsCopy Anchor Link

A five-token sentence gets token probabilities 0.5, 0.4, 0.9, 0.7 and 0.6. The sentence probability is 0.5 × 0.4 × 0.9 × 0.7 × 0.6 = 0.0756. Now imagine 1,000 tokens: the product would be a number with hundreds of zeros after the decimal point, too small for a computer to store. Take logs instead. ln 0.5 = -0.693, ln 0.4 = -0.916, ln 0.9 = -0.105, ln 0.7 = -0.357, ln 0.6 = -0.511. Their sum is -2.582, and e raised to -2.582 is 0.0756, the same answer. Sums of logs stay a comfortable size, so training adds logs instead of multiplying probabilities. The loss you meet later is "minus the log of the probability given to the right token": 0 when the model was certain and right, and large when it gave the right token a tiny chance.

What each step of a log meansCopy Anchor Link

A log counts how many times you scaled something down. Natural logs of small probabilities: ln 0.5 = -0.693, ln 0.1 = -2.303, ln 0.01 = -4.605, ln 0.001 = -6.908. Each time the probability shrinks by a factor of 10, the log drops by 2.303. A number that gets tiny stays a manageable size.

A second example: comparing sentences of different lengthCopy Anchor Link

Sentence A has 5 tokens with probability 0.0756, from the example above. Sentence B has 10 tokens, each with probability 0.7, so its probability is 0.7¹⁰ = 0.0282. Raw probability says A is "better", but only because it is shorter, since every extra token multiplies in another number below 1. Compare the average log per token instead. A: -2.582 / 5 = -0.516. B: 10 × ln 0.7 = -3.567, and -3.567 / 10 = -0.357. B is closer to 0, so B has the higher per-token confidence. This is why scores for language models are usually averaged per token.

When logs are not needed, and what goes wrongCopy Anchor Link

For short products such as the 5-token sentence, plain multiplication is exact enough, and logs add nothing. They matter when the product is long. A 64-bit decimal number cannot go below about 10⁻³⁰⁸. With 1,000 tokens each at probability 0.1, the product is 10⁻¹⁰⁰⁰, and a computer stores 0.0. The sum of logs is 1,000 × (-2.303) = -2,303, which is fine. The other trap is probability 0: ln 0 has no value (it heads to minus infinity), so a loss of "inf" or "nan" in training often means some token got exactly zero probability. Check by printing the smallest probability in the batch.

Check it by handCopy Anchor Link

Take three tokens with probabilities 0.5, 0.25 and 0.125. The product is 0.015625. The logs are -0.693, -1.386 and -2.079, which add to -4.159. Raise e to that power and you get 0.0156 again. Same answer by two routes.

Common mistakeCopy Anchor Link

Thinking a logarithm makes numbers bigger. It compresses them: log₂ of a million is only about 20. Also check the base. log₂ gives bits, ln gives nats, and the idea is the same.