Sampling parameters
A handful of settings control how the next token is chosen and when the answer stops. Knowing them lets you tune behavior without changing the prompt.
Words on this page (8)
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- prompt
- : The input text you give a model, including instructions, examples and data.
- temperature
- : A setting that sharpens or flattens the probabilities before sampling. Low is predictable, high is varied.
- top-p
- : A sampling rule that only considers the smallest set of tokens whose probabilities add up to p.
- sampling
- : Picking the next token at random according to the model's probabilities.
- JSON
- : A text format for structured data: keys and values in braces, like {"name": "Ada"}.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
The settings
rescales probabilities: near 0 is repeatable, higher is more varied. Top-k keeps only the k most likely , for example 40, and samples among them. (nucleus ) keeps the smallest set of tokens whose probabilities add up to p, for example 0.9, so the set grows when the model is unsure and shrinks when it is confident. Repetition or frequency penalties lower the score of tokens that already appeared, reducing loops. Max tokens caps the answer length. Stop sequences end generation when a given string appears. A seed makes sampling repeatable on providers that support it.
Example
Probabilities for the next token: A 0.50, B 0.25, C 0.15, D 0.07, E 0.03. Top-k 2 keeps A and B and rescales them to 0.67 and 0.33. Top-p 0.9 keeps A, B and C (0.50 + 0.25 + 0.15 = 0.90) and drops D and E.
Rules of thumb
Extraction, classification and code: temperature 0 to 0.3. Brainstorming and creative writing: 0.8 to 1.0. Change either temperature or top-p, not both. Some reasoning models fix these values and ignore your settings. Even at temperature 0, output can differ between calls on some providers, so do not rely on bit-exact repeatability.
A worked example: same prompt, three settings
: "Name a fruit." Temperature 0: "Apple" every time. Temperature 0.7: apple, banana, mango across runs. Temperature 1.5: occasionally "dragonfruit" or something odd. For a form that must name exactly one valid fruit from a list, use 0. For brainstorming ten product names, use 0.9.
How the settings stack
Settings act in sequence, and earlier ones change what later ones see. Reuse the probabilities A 0.50, B 0.25, C 0.15, D 0.07, E 0.03. Dividing the scores by a temperature of 0.5 is the same as squaring each probability and rescaling so they sum to 1. The squares are 0.25, 0.0625, 0.0225, 0.0049 and 0.0009, which total 0.3408. The new probabilities are about 0.734, 0.183, 0.066, 0.014 and 0.003.
Now apply top-p 0.9. A and B together reach 0.917, so only A and B survive. Earlier, top-p 0.9 alone kept three tokens. Lowering temperature shrank the candidate set as well. The order can differ between libraries and services, so test rather than assume.
Trade-offs and when not to use them
Low temperature gives repeatable output but can loop and sound flat. High temperature gives variety but more errors. Top-k uses the same cutoff whether the model is sure or unsure, which top-p handles better. A seed helps you debug, but it is not a guarantee of identical output across providers or versions.
Do not tune sampling to fix a bad prompt. If answers are wrong because the instructions are unclear, no setting repairs that.
What goes wrong
Match the symptom to the dial.
- cut off halfway: max tokens is too low. Check why generation stopped.
- An answer ends too early: a stop sequence appears inside the content.
- Odd or unrelated words: temperature is too high.
- The same line repeated: temperature is too low, or no repetition penalty is set.
the settings with every request. Then rerun the failing prompt about 10 times, changing one dial at a time.
Try it yourself
In any playground that exposes temperature, send "Name a fruit." ten times at 0 and ten times at 1. Count distinct answers. Expect very few at 0, usually one or two, and noticeably more at 1.
Common mistake
Turning both temperature and top-p up at once, which compounds the randomness and produces nonsense. Change one dial, test, then decide.
The math, when you are ready
Temperature is one number inside the formula. See it with real numbers in Probability and softmax.
A chef follows a recipe the same way every time for a customer who wants consistency, and improvises when a customer asks for something surprising. Sampling settings are the model's "how much to improvise" dials. For a tax form you want the first kind of chef. For a poem you might want the second.
Check yourself
Token probabilities are A 0.50, B 0.25, C 0.15, D 0.07, E 0.03. Which does top-p 0.9 keep?
Not quite. 0.50 + 0.25 + 0.15 reaches 0.90, so D and E are dropped.
What does a stop sequence do?
Not quite. It cuts the output at a chosen marker, such as the end of a field.
You want ten different product name ideas. Which setting fits best?
Not quite. Brainstorming benefits from variety, which higher temperature provides.
You got 0 of 3 right.
Go deeper
- Docs
Anthropic · Temperature, top-p, top-k, max tokens and stop sequences.
- Article
Hugging Face · Greedy, beam, top-k and top-p, with code.
- Video
Andrej Karpathy · Sampling is built into the generation loop near the end.
Spotted a mistake or something unclear? Report it on GitHub.