Evaluating RAG
Measure retrieval and generation separately, so you know which one to fix.
Words on this page (11)
- retrieval
- : Finding the most relevant pieces of text for a query.
- chunk
- : A small piece of a document, sized for indexing and retrieval.
- recall
- : Of the truly positive cases, the share the system found.
- precision
- : Of the cases a system flagged, the share that were truly positive.
- MRR
- : Mean reciprocal rank: the average of 1 divided by the rank of the first correct result. Higher is better.
- rubric
- : A written list of criteria used to grade an output.
- embedding
- : A list of numbers that represents the meaning of a piece of text or an image, so similar things sit close together.
- reranker
- : A second model that rescores a short list of search results by reading query and passage together.
- latency
- : The time you wait for a response.
- prompt
- : The input text you give a model, including instructions, examples and data.
- calibration
- : Whether a model's confidence matches how often it is right. A well calibrated model is right about 90 percent of the time when it says 0.9.
Retrieval metrics
Build a set of questions, each with the or document that answers it. at k: how often the right chunk is in the top k results. at k: what fraction of the top k are relevant. (mean reciprocal rank): the average of 1 / rank of the first right result, so rank 1 scores 1 and rank 4 scores 0.25. Example: ranks of 1, 4 and 2 across three questions give (1 + 0.25 + 0.5) / 3 = 0.58.
Generation metrics
Faithfulness: is every claim in the answer supported by the retrieved chunks? Answer relevance: does it address the question? Context relevance: were the retrieved chunks needed? These are usually scored by a model judge with a , calibrated against a sample you labeled yourself. RAGAS and DeepEval implement these metrics.
The loop
Change one thing, such as chunk size, model or , and rerun both sets of metrics. Keep the change only if it helps enough to justify the added cost or .
A worked example: scoring retrieval
Five test questions. The right chunk is ranked 1st, 3rd, not found in the top five, 1st and 2nd. Recall at 5 is 4 out of 5 = 0.8. Reciprocal ranks: 1, 0.33, 0, 1, 0.5, so MRR = 2.83 / 5 = 0.57. Change the chunk size and rerun. If recall rises to 0.9, keep it.
Building the test set
Start with 30 to 50 questions. Take them from real user logs if you have them, and write the rest by hand. For each, record the chunk or page that answers it. Include some questions the documents cannot answer, with the expected result "I do not know". Cover the types your users ask: exact lookups, vague questions and follow-ups.
A second example: scoring faithfulness
Faithfulness checks each claim in an answer against the retrieved chunks. The chunk says "Employees receive 25 days after 12 months." The answer "25 days" has one claim, and it is supported: 1 out of 1. The answer "25 days, plus 5 sick days" has two claims, and only the first is supported: 1 out of 2, a faithfulness of 0.5. The was fine here. The generation step added something unsupported, so the fix is in the , not the index.
What goes wrong
Small test sets are noisy. With 10 questions, one question is 10 percentage points, so 0.8 against 0.9 may be a single flip. With 50 questions, one is 2 points. Do not celebrate a small gain on a small set.
Tuning on the same questions you score on overfits: you pick settings that suit those questions and nothing else. Keep some questions aside as a final check. A model judge can also be wrong or favor longer answers, so read a sample of its verdicts yourself.
Trade-offs
More questions mean more reliable numbers but more labeling work. Model judges are cheap and fast, but need . Human review is slow and trustworthy. A practical split is a model judge on every change and human review on a small sample.
Common mistake
Changing three things at once. You will not know which one helped.
A team improves their prompt and the demo answers look better. Two weeks later support complains of more wrong answers. They had been judging by a handful of examples. A set of fifty questions with known sources would have shown that retrieval got worse while the wording got nicer.
Check yourself
The right answer ranks 1st, 4th and 2nd for three questions. What is the MRR?
Not quite. (1 + 0.25 + 0.5) / 3 is about 0.58.
Why evaluate retrieval separately from generation?
Not quite. A wrong answer can come from either, and the fix differs.
You change the chunk size and the reranker at once and recall rises. What is the problem?
Not quite. Change one thing at a time so you know what to keep.
You got 0 of 3 right.
Go deeper
- Repo
Exploding Gradients · Metrics for retrieval and generation quality.
- Repo
Confident AI · Unit-test style evals for LLM apps, including RAG metrics.
- ArticleYour AI product needs evalsArticle
Hamel Husain · The look-at-your-data loop that applies to RAG.
- DocsRagas docsDocs
Ragas · Metrics for faithfulness, context precision and recall.
- DocsDeepEval docsDocs
Confident AI · Test-case style evaluation for RAG and agents.
Spotted a mistake or something unclear? Report it on GitHub.