Games and reinforcement learning

What happenedCopy Anchor Link

Deep Blue (1997) beat the chess champion with brute-force search. AlphaGo (2016) beat Lee Sedol at Go using neural networks to guess good moves and positions, trained first on human games and then by playing itself. In game two, its move 37 surprised experts as something no human would play. AlphaZero (2017) learned chess, shogi and Go from scratch with only the rules, by playing itself, and beat the best programs.

The idea: reinforcement learningCopy Anchor Link

An agent takes actions in an environment and receives rewards. It learns a policy, a rule for choosing actions, that maximizes total reward. There are no labels, only a score that arrives late. The same idea later trained language models: with a reward for correct math or passing code tests, models learned to reason in longer chains of thought.

Why it mattersCopy Anchor Link

Self-play and reward signals let a system exceed its teachers. The limit is that you need a reward you can compute. That is why reinforcement learning works best on games, math and code, where answers can be checked.

A worked example: learning a tiny game by rewardCopy Anchor Link

A robot in a 3-cell corridor starts in cell 1. Cell 3 holds a reward of +10. Moving costs -1. At first it moves randomly. After a few hundred tries, it notices that moving right from cell 2 usually leads to +10 and records a high value for that action. Soon the value spreads backward: moving right from cell 1 is good too, because it leads to cell 2. This is the core idea of Q-learning.

How the value spreads: a traceCopy Anchor Link

Q-learning keeps a score for each action in each cell. After every move it updates that score with a simple rule: new score = old score + learning rate × (reward + best score of the next cell − old score). These are invented numbers for illustration. Use a learning rate of 0.5, all scores starting at 0, and a net reward of +9 for the move that reaches cell 3 (+10 minus the move cost of 1).

1. From cell 2 the robot moves right and reaches the goal. The score for "right in cell 2" becomes 0 + 0.5 × (9 + 0 − 0) = 4.5.
2. From cell 1 it moves right to cell 2, paying -1. The score for "right in cell 1" becomes 0 + 0.5 × (-1 + 4.5 − 0) = 1.75.
3. Next time it reaches the goal from cell 2, that score becomes 4.5 + 0.5 × (9 − 4.5) = 6.75. Each pass moves the scores toward 9 for cell 2 and 8 for cell 1. Good news travels backward one cell per visit.

Explore or exploitCopy Anchor Link

The agent must choose between the best action it knows and a new one. If it always picks the best known action, it can lock onto a mediocre habit and never find a better route. A common fix is to try a random action a small share of the time. Too much random play wastes effort. Too little leaves better routes undiscovered. In the corridor this is trivial. In Go the number of possible positions is so large that the balance matters a great deal.

What goes wrongCopy Anchor Link

A reward that arrives only at the end of a long game gives the agent almost nothing to learn from early on. The symptom is a score curve that stays flat for a long time. Adding small rewards along the way helps, but each one is a new chance to be gamed. Another symptom is a rising score with strange behavior. Confirm it by watching replays of the agent, not only the score chart.

Try it yourself: three more updatesCopy Anchor Link

On paper, repeat the trace above for the cell 2 score. Apply the rule again and again. You should get 4.5, then 6.75, then 7.875. Then update the cell 1 score once using the new cell 2 value. The numbers climb toward 9 and 8 but never jump there at once. Ten minutes is enough, and it shows why learning takes many tries.

Common mistakeCopy Anchor Link

Expecting the same method to work where there is no clear score. Defining the reward is the hard part. A badly chosen reward gets exploited: a cleaning robot rewarded for "no visible mess" may learn to hide the mess.