LLM as judge
A model can grade outputs at scale, but only after you check the grader against human judgments.
Words on this page (4)
- prompt
- : The input text you give a model, including instructions, examples and data.
- rubric
- : A written list of criteria used to grade an output.
- data drift
- : Real-world inputs slowly changing from what a system was tested on, so quality falls with no code change.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
What it is
You a model with the input, the output and a , and ask for a verdict. It is useful for questions code cannot answer, such as whether a summary is faithful to its source. Prefer a binary pass or fail on one narrow criterion over a one to ten score, since scores are noisy and hard to act on.
Make it trustworthy
Label a sample yourself. Run the judge on the same sample. Measure agreement, and look at the cases where they differ, then fix the rubric. Use a separate model or prompt from the one that generated the output, since models are lenient toward their own work. Keep checking agreement over time, because the system and the .
A worked example: a narrow rubric
Question for the judge: "Does the answer contain any claim that is not supported by the source text? Reply PASS or FAIL and quote the unsupported claim if any." Run it on fifty answers you have already labeled. If it agrees on 46 of 50, look at the four disagreements and adjust the rubric. Then run it on thousands.
Biases to watch
Judges prefer longer answers, prefer their own style, and can be swayed by confident tone. Use a different model than the one that wrote the answer, compare two answers in both orders, and keep spot-checking.
Agreement can fool you
Suppose you label 50 answers and 45 are good, 5 are bad. A judge that always says PASS agrees with you on 45 of 50, which is 90 percent. It catches nothing.
So also count the cases that matter. Of the 5 bad answers, how many did the judge mark FAIL? If it caught 2, it found 40 percent of the failures, however high the overall agreement looks. Check both numbers.
Testing for order bias
When a judge compares two answers, the position can sway it. To measure this, run each pair twice with the order swapped. Invented numbers: of 20 pairs, the judge picks the same answer both ways in 12. The other 8 flipped with order, so 40 percent of its verdicts are partly noise.
Fixes: count a pair as a win only when the judge picks the same answer in both orders, and call the rest ties.
When a judge is the wrong tool
A judge cannot check facts it cannot see. Asked "is this answer correct?" with no source, it guesses from its own memory. Give it the source text or a reference answer, or use a code check if the answer is exact, such as a date or a number.
Common mistake
Scores from 1 to 10. They are noisy and hard to act on. Pass or fail on one criterion is clearer.
A teacher cannot read ten thousand essays, so she trains an assistant to grade them with a checklist. First she grades fifty herself, then compares with the assistant's grades and fixes the checklist where they disagree. A model judge works the same way: it scales you up, but only after you calibrate it.
Check yourself
How do you know your judge is any good?
Not quite. Validate the grader against human judgments before trusting its scores.
Why prefer pass or fail to a 1 to 10 score?
Not quite. Narrow binary criteria give a clearer signal.
The judge and the writer are the same model. What bias should you expect?
Not quite. Models tend to favor their own outputs, so use a different judge and spot-check.
You got 0 of 3 right.
Go deeper
- Article
Hamel Husain · Binary judgments and calibrating against a domain expert.
- Paper
Zheng et al. · Measures judge agreement with humans and documents biases.
- Repo
OpenAI · Framework and registry of evals.
Spotted a mistake or something unclear? Report it on GitHub.