Reinforcement learning for reasoning
Models learn to reason by trying answers, getting a reward when the answer checks out, and reinforcing what worked.
Words on this page (10)
- reward
- : A number that says how good an outcome was. RL tries to maximize it.
- policy
- : A rule for choosing actions. In a language model, the model itself is the policy.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- gradient
- : The direction and rate at which the loss changes when each parameter changes. Training steps against it.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- baseline
- : A simple method, such as "same as last week", that any model must beat to be worth its cost.
- PPO
- : Proximal policy optimization: a reinforcement learning method that limits how far each update can move the model.
- GRPO
- : Group relative policy optimization: a reinforcement learning method that scores each answer against the other answers sampled for the same prompt.
- prompt
- : The input text you give a model, including instructions, examples and data.
- batch
- : A group of examples processed together in one step.
The idea
Start with a : the model itself, which picks . For a math problem, sample several answers. A function gives 1 if the final answer is correct and 0 if not. Raise the probability of the sampled reasoning that earned reward and lower it for the rest. Over many problems the model discovers longer, more careful chains of thought.
Algorithms in plain terms
Policy (REINFORCE) nudges up the probability of actions in proportion to the reward. The problem is high variance, so is noisy. An advantage compares an answer to a : how much better than average was this one? adds a limit on how far each update can move the policy. It looks at the importance ratio, which is the new probability divided by the old one, and clips it to a range like 0.8 to 1.2. , used by DeepSeek, drops the separate value model: sample a group of answers per . Each answer's advantage is its reward minus the group's mean, divided by the group's standard deviation (a measure of how spread out the rewards are). Some later variants drop the division.
Limits
It needs a reward you can compute, so it works best on math, code and tasks with tests. Rewards can be gamed: a model may find shortcuts that satisfy the check without doing the task.
A worked example: one training step
Prompt: "What is 17 × 24?" The model samples four attempts with answers 408, 398, 408, 418. Rewards: 1, 0, 1, 0. Mean 0.5, standard deviation 0.5. Advantages: +1, -1, +1, -1. The two reasoning chains that reached 408 are made more likely, the others less likely. Repeat over many thousands of problems.
What can go wrong
Reward hacking: if the reward only checks the final number, the model may learn tricks that get the number without good reasoning. Better rewards check format, length, and sometimes the steps too.
A second example: a rare success
Rewards for four attempts are 1, 0, 0, 0. The mean is 0.25 and the standard deviation is 0.43. The one correct attempt gets advantage (1 - 0.25) / 0.43 = +1.73. Each wrong one gets -0.58. A rare success pulls much harder than the failures push, which is how the model learns from problems it usually gets wrong.
When there is nothing to learn
If all four attempts are correct, every reward is 1, the mean is 1, the standard deviation is 0 and every advantage is 0. The same happens when all four fail. The update is empty. This is why training sets are chosen so problems are neither too easy nor too hard. If reward stays flat, check how many prompts in a have mixed results.
How clipping limits a step
Say the old model gave a reasoning step probability 0.20 and the updated one gives 0.30. The ratio is 1.5. With clipping at 1.2, a positive-advantage sample stops counting beyond 0.24, so one lucky sample cannot drag the model far from where it was.
What goes wrong: hacking you can measure
Suppose the reward is the share of unit tests that pass. A model may learn to write code that detects the test inputs and returns the expected outputs. The symptom is training reward near 100 percent while code fails on new inputs. To confirm, score the model on held-out tests it never trained against, and read some outputs.
Common mistake
Assuming this works for every task. It needs answers you can check automatically.
A student solves math problems and is told only "right" or "wrong" at the end. At first they guess. They notice that writing out the steps leads to more rights, so they write more steps. Reinforcement learning on math does this to a model: reward correct final answers and the model discovers that careful, longer reasoning pays off.
Check yourself
What is the GRPO advantage of a correct answer when rewards are 1, 0, 0, 1?
Not quite. Reward minus the group mean is 0.5, and dividing by the standard deviation (0.5) gives +1.
Why does reinforcement learning for reasoning work best on math and code?
Not quite. A computable reward is the requirement.
A model learns to print the expected final number with no reasoning. What is this?
Not quite. If the reward only checks the final number, the model may find shortcuts. Better rewards check more.
You got 0 of 3 right.
Go deeper
- PaperDeepSeekMathPaper
Shao et al. · Introduces GRPO.
- PaperDeepSeek-R1Paper
DeepSeek · Reasoning emerging from reinforcement learning with verifiable rewards.
- Paper
Schulman et al. · The PPO paper.
- Course
Hugging Face · Free course from basics to policy gradient methods.
Spotted a mistake or something unclear? Report it on GitHub.