Reranking search results

What it isCopy Anchor Link

Retrieval is usually two stages. Stage one embeds the question and each passage separately and compares the vectors. It is very fast, so it can scan millions of passages, but a single vector cannot hold every detail. Negations, numbers and conditions often get lost. Stage two takes the top 50 or so candidates and scores each one with a cross-encoder, a model that reads the question and the passage together. It is slower and much sharper. You keep the best 5 and pass those to the model.

A worked exampleCopy Anchor Link

Question: "Can I return a gift card?" Stage one finds three passages with embedding similarity scores: B "Returns are accepted within 30 days" (0.78), C "Gift card balances never expire" (0.74) and A "Gift cards cannot be refunded" (0.71). Embeddings put B first because it matches "return". The reranker reads each with the question and scores A 0.93, B 0.41 and C 0.20. The right passage is now first. The model answers "No, gift cards cannot be refunded", which stage one alone would likely have missed.

Using oneCopy Anchor Link

Fetch 30 to 100 candidates, rerank them, keep 3 to 10. The reranker adds time, often tens to a few hundred milliseconds depending on the model and the number of passages, and some cost. Measure it: run your test questions and compare recall at 5 before and after, as in the evaluating RAG lesson. Keep the reranker only if the gain is worth the delay.

Why it cannot be the first stageCopy Anchor Link

In stage one, every passage was turned into a vector in advance. At query time you embed only the question and compare. A cross-encoder cannot do this, because its score depends on the question and the passage together. It must run once per pair, at query time.

Invented numbers: if one run takes 4 ms, reranking 50 candidates costs 50 × 4 = 200 ms. Reranking a million passages would take 4,000,000 ms, over an hour. So stage one narrows the field and the reranker only sees the shortlist.

What goes wrongCopy Anchor Link

A reranker can only reorder what it is given. If the right passage is not among the 50 candidates, no reranker can find it. The symptom is a top result that looks plausible but is wrong. To confirm, check whether the correct passage was in the candidate list at all. If it was missing, fix stage one (chunking, hybrid search, query rewriting), not the reranker.

Long passages are another trap. Rerankers have a maximum input length, and text past it is usually cut off, so a key sentence at the end of a long chunk may never be read.

A second example: when it adds nothingCopy Anchor Link

Question: "Where is the Paris office?" Stage one already ranks the passage "Our Paris office is at 12 Rue Example" first, and the second place is far behind. A reranker would confirm the order and add 200 ms. Reranking helps on questions where several passages look alike, such as the gift card case above. Test on your own question mix to see how many are that hard.

Trade-offsCopy Anchor Link

You gain accuracy on the final few results. You pay in delay, in hosting or per-call cost, and in one more moving part. Fetch more candidates for better recall, but each extra candidate adds time. A common starting point is 50 candidates in, a handful out.

Common mistakeCopy Anchor Link

Reranking only the 5 results you already planned to use. There is nothing to reorder. The point is to search wide, then narrow.