Next-token prediction and sampling

What it isCopy Anchor Link

Given the tokens so far, the model outputs a probability for every token in its vocabulary (its full list of known pieces). A sampler picks one, appends it, and the model runs again. An answer is just this loop repeated until the model outputs a special end-of-answer marker called a stop token.

Temperature reshapes the probabilities before sampling (it divides the scores first, shown below). Near 0 the top token wins almost every time, so output is stable. Higher values flatten the odds, so unlikely tokens get picked more often. Top-p keeps only the smallest set of tokens whose probabilities add up to p, then samples inside it. For example, with probabilities 0.66, 0.24 and 0.10 and p = 0.8, the first two add up to 0.90, so the third token is dropped and the other two are rescaled to 0.73 and 0.27.

Try itCopy Anchor Link

Use the sampler above the lesson, which shows three tokens and their odds. Raise the temperature and watch how the picks spread out. Then set it near zero and sample ten times.

Worked example: one stepCopy Anchor Link

Prompt: "The capital of France is". The model scores possible next tokens: "Paris" 92 percent, "a" 2 percent, "Lyon" 1 percent, and so on. The sampler picks one, usually "Paris". Now the prompt is "The capital of France is Paris" and the model predicts the next token again, maybe a period. Repeat until it predicts an end marker.

Why this explains the weirdnessCopy Anchor Link

The model never plans the whole answer. That is why it can start a sentence confidently and then box itself into an ending that does not fit.

Inside one step, with small numbersCopy Anchor Link

The model first produces a raw score for each token, called a logit. Softmax turns scores into probabilities that sum to 1. Invented scores for three tokens: 2, 1 and 0. Softmax raises e (about 2.718) to each score, which gives 7.389, 2.718 and 1, then divides each by their total of 11.107. The probabilities are about 0.665, 0.245 and 0.090.

Temperature divides the scores first. At temperature 0.5 the scores become 4, 2 and 0, and the probabilities sharpen to about 0.867, 0.117 and 0.016. At temperature 2 they become 1, 0.5 and 0, and flatten to about 0.506, 0.307 and 0.186. Same model, same prompt, different odds.

What goes wrong: loopsCopy Anchor Link

At low temperature a model can repeat itself. After a phrase has appeared twice, the most likely next token is often the start of the same phrase again, and the loop feeds itself. The symptom is the same sentence repeated until the length limit.

To confirm, rerun with a higher temperature or a repetition penalty, a setting that lowers the score of tokens already used. If the loop disappears, sampling was at least part of the cause.

Answer order mattersCopy Anchor Link

Because tokens are committed one at a time, the first words steer the rest. If a reply must begin "Yes", the model often ends up justifying yes even when the evidence says no. Asking for the reasoning first and the verdict last lets the verdict rest on that reasoning.

Test it on a yes or no question with a tricky answer. Run it once asking for the verdict first and once asking for reasoning first. Compare the results over about 10 runs each.

How to measure variabilityCopy Anchor Link

For a task you care about, run the same prompt 10 times at your chosen settings and count the distinct answers. One or two means the task is stable. Many means the model is unsure, and a clearer prompt or lower temperature should come before anything else.

Common mistakeCopy Anchor Link

Believing the model "looks up" facts. It predicts likely text. When a fact was common in training, the likely text is right. When it was rare, the likely text can be a convincing guess.

The math, when you are readyCopy Anchor Link

The step that turns scores into probabilities is called softmax. It has a worked example with small numbers in Probability and softmax. You do not need it to use models.