Code and math

CodeCopy Anchor Link

GitHub Copilot (2021) brought autocomplete from large models into editors. Chat assistants followed, then agents that read a repository, edit files and run tests. SWE-bench measures this on real GitHub issues: the model gets the repository and the issue and must make the tests pass. Scores climbed from low single digits in 2023 to a majority of tasks solved on its verified subset in 2025.

MathCopy Anchor Link

Reasoning models trained with reinforcement learning on problems with checkable answers improved quickly on competition math. In 2024 DeepMind's AlphaProof and AlphaGeometry reached silver-medal level on International Mathematical Olympiad problems. In 2025 several systems reached gold-medal level.

Why it mattersCopy Anchor Link

When you can verify a result automatically, you can train on it and you can trust it more. That is why coding agents lead the agent field, and why building verification into your own tasks pays off.

A worked example: why checkable work improves fastestCopy Anchor Link

Suppose a model attempts 1,000 coding problems with hidden tests. Each attempt either passes or fails, with no argument. That pass/fail signal can be used to train the model: reinforce the attempts that passed. For an essay there is no such signal, only opinions. Domains with automatic checks, like code, math and games, therefore improve first.

What benchmarks tell youCopy Anchor Link

SWE-bench uses real issues from open-source projects. A model is scored on how many it resolves with all tests passing. The number rose quickly from 2023 to 2025, which tells you the trend, but your own codebase is different. Always try a candidate tool on your own code.

How a test becomes a training signalCopy Anchor Link

The loop is simple enough to follow step by step, using invented numbers.

1. Give the model a problem and keep the tests hidden.
2. Ask for 8 different attempts.
3. Run every attempt against the tests.
4. Score each one 1 for pass and 0 for fail. Suppose 3 pass, so the pass rate is 3 ÷ 8 = 37.5 percent.
5. Adjust the model so the 3 passing styles become more likely. Repeat over thousands of problems. No human has to read the attempts, which is why this scales.

A second example: where checking is weakCopy Anchor Link

Ask a model to write a friendly welcome email. There is no test that says the email is good. Even in code, tests can be weak. If the tests only check one input, a function that hardcodes the expected answer passes. The symptom is green tests and wrong behavior on real data. Confirm it by adding a test the model never saw, with a different input, and reading the diff for special-case code.

Reading a benchmark honestlyCopy Anchor Link

A high score on a public test shows the trend, but three things can inflate it. The problems may have leaked into the training data, so the model remembers answers. The test may cover only some kinds of work, such as small bug fixes. The agent may have been given many attempts. Ask what a score measures and how many tries were allowed before you compare tools.

Try it yourself: tests firstCopy Anchor Link

Pick a small function, such as one that applies a discount to a price. Before asking a chat model to write it, write five tests: three normal cases and two edge cases, such as a price of zero and a negative discount. Paste the description, run the tests on the answer and count passes. Models often pass the normal cases and miss an edge case. The lesson is that your tests, not the model's confidence, decide whether the code is right.

Common mistakeCopy Anchor Link

Accepting generated code because it compiles. Read it, run the tests and look for unrequested changes.