Probability and softmax
A model outputs raw scores. Softmax turns them into probabilities that add up to 1, and temperature controls how sharp they are.
Temperature and softmax
picked 0
picked 0
picked 0
Pick the next token
Low temperature: the top score almost always wins. High: choices spread out.
Words on this page (9)
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
- temperature
- : A setting that sharpens or flattens the probabilities before sampling. Low is predictable, high is varied.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- logits
- : The raw scores a model produces for each possible next token, before softmax.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- top-p
- : A sampling rule that only considers the smallest set of tokens whose probabilities add up to p.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- sampling
- : Picking the next token at random according to the model's probabilities.
The idea
The model's last layer produces one number per possible next , called . converts them: p(i) = e^(zᵢ/T) / Σⱼ e^(zⱼ/T). Exponentiating makes everything positive. Dividing by the sum makes them add to 1. T is the . T = 1 is the plain version.
Example
Logits for three tokens: [2.0, 1.0, 0.1]. At T = 1, e² = 7.39, e¹ = 2.72, e^0.1 = 1.11. The sum is 11.21. Probabilities are 0.66, 0.24, 0.10. At T = 0.5 the logits double to [4, 2, 0.2], giving e⁴ = 54.6, e² = 7.39, e^0.2 = 1.22, sum 63.2, so probabilities 0.86, 0.12, 0.02. Lower temperature sharpens the choice. At T = 2 the logits halve to [1, 0.5, 0.05] and the picks flatten to about 0.50, 0.30, 0.19.
Why it matters
The same function turns scores into and a classifier's scores into class probabilities. Temperature and are knobs on this distribution.
Try it
Move the sliders in the demo. Watch how temperature reshapes the bars, then sample ten tokens and see how often the top one wins.
Why the exponential
Raw scores can be negative or huge. The exponential e^x makes every score positive and makes bigger scores stand out more. Dividing by the total then makes them add to 1. A score of 2.0 versus 1.0 becomes 7.4 versus 2.7, so the leader gets a clearly bigger share.
How it works step by step
Softmax has three steps, shown here for logits [1, 2, 3] at T = 1.
- Exponentiate each score: e¹ = 2.718, e² = 7.389, e³ = 20.086.
- Add them: 2.718 + 7.389 + 20.086 = 30.193.
- Divide each by the total: 0.090, 0.245, 0.665. They add to 1. Notice that the gap matters, not the ratio. Logits [3, 0] give 0.953 and 0.047. Logits [6, 0] give 0.9975 and 0.0025. Logits [2, 2, 2] give one third each, whatever the temperature.
A second example: why code subtracts the biggest score
Real code first subtracts the largest logit from every logit. The answer does not change, because the same factor cancels on top and bottom. Take logits [1000, 999]. Computing e¹⁰⁰⁰ overflows, meaning the number is too big to store, and the program returns infinity. After subtracting 1000 the scores are [0, -1], so e⁰ = 1 and e⁻¹ = 0.368. The probabilities are 1 / 1.368 = 0.731 and 0.269, the same as for [1, 0]. If you ever see nan from your own softmax, this is the first thing to check.
Trade-offs of temperature and top-p
Lower temperature gives repeatable text but can loop and repeat itself. Higher temperature gives variety but more odd words. At T = 0 the formula would divide by zero, so treat it as "always pick the top token". Top-p cuts the tail instead: keep the smallest group of top tokens whose probabilities add to at least p. With [0.66, 0.24, 0.10] and p = 0.8, the first two add to 0.90, so the third is dropped. Re-dividing by 0.90 gives 0.73 and 0.27. Temperature reshapes every probability, while top-p removes unlikely tokens.
Check it by hand
Predict before you calculate: for logits [0, 0, 1], which token wins and by how much? Then compute e⁰ = 1, 1 and e¹ = 2.718, with total 4.718. The probabilities are 0.212, 0.212 and 0.576. The top token wins, but it does not take everything, because a gap of 1 is small.
Common mistake
Thinking temperature 0 gives the "correct" answer. It gives the most likely answer, which can still be wrong. Low temperature gives consistency, not truth.
A weather app does not say "it will rain". It says "70 percent chance". A language model does the same for every next word: "the" 12 percent, "a" 9 percent, and so on. Softmax is the little formula that turns the model's raw scores into those percentages, and temperature is the dial that makes it bolder or more careful.
Check yourself
What does lowering the temperature do to the output distribution?
Not quite. Dividing the scores by a small T makes the differences bigger, so the leader dominates.
Softmax outputs always do which of these?
Not quite. That is what lets them be read as probabilities.
Scores are [2, 1, 0.1] at T = 1 and give 66 percent for the first token. At T = 2, what happens to that 66 percent?
Not quite. Higher temperature flattens the distribution. The worked example gives about 50, 30 and 19 percent.
You got 0 of 3 right.
Go deeper
- VideoArgMax and SoftMaxVideo
StatQuest · Step-by-step with numbers.
- ArticleSoftmax functionArticle
Wikipedia · Definition and properties, including the temperature form.
Spotted a mistake or something unclear? Report it on GitHub.