How to read benchmarks and leaderboards

What to checkCopy Anchor Link

Is the test saturated, with top models all near 100 percent so differences are noise? Could the model have seen the answers in training (contamination)? Does the task look like yours? Was the score measured by the vendor or independently? How many runs, and what was the spread? Human preference arenas measure style and helpfulness in chat, which is not the same as correctness on your task. Benchmarks for agents depend heavily on the harness used, so a score is for a model plus a harness.

How to use themCopy Anchor Link

Use leaderboards to shortlist, then run your own examples. Track a small number of benchmarks that resemble your work. Be suspicious of any single headline number.

A worked example: is the gap real?Copy Anchor Link

Model A scores 87.3 and model B 86.9 on a test with 500 questions. With that many questions the typical sampling error is about 1.5 points (about 3 points at 95 percent confidence), so a gap of 0.4 is within noise. Treat them as equal until a larger or different test separates them.

How the noise estimate worksCopy Anchor Link

A score on n questions is a sample. The typical sampling error is sqrt(p(1 - p) / n). For p = 0.87 and n = 500 that is sqrt(0.87 × 0.13 / 500), about 0.015, or 1.5 points.

A gap between two scores is noisier than either score alone, because both carry error. Treating them as independent, the gap error is about 1.5 times the square root of 2, or about 2.1 points. About 95 percent of the time the true gap sits within roughly twice that. A 0.4 gap is far inside it. Models tested on the same questions have a tighter gap in practice, but 0.4 still tells you very little.

What goes wrong: a model that has seen the testCopy Anchor Link

Contamination means test questions appeared in the training data. The symptom is a high public score that does not carry over. A model scores 90 on a famous test, then does much worse on questions you wrote in the same style.

To check, write 20 fresh questions of the same type and compare. A large drop suggests memorization. It does not prove it, since your questions may also be harder.

When a public benchmark is the wrong toolCopy Anchor Link

You summarize legal contracts. A coding leaderboard tells you almost nothing about that. Build your own test:

1. Collect 30 real inputs from your work.
2. Write the answer you want for each, or a short rubric (a checklist for grading).
3. Run each candidate model three times on all 30.
4. Record the average and the spread between runs.

If a model's three runs vary by more than the gap between models, you cannot separate them yet. Add more examples.

Common mistakeCopy Anchor Link

Quoting a single number without its context.