Advanced RAG techniques
Once basic retrieval works, these techniques fix the specific failures you measure.
Words on this page (13)
- retrieval
- : Finding the most relevant pieces of text for a query.
- hybrid search
- : Combining keyword search and vector search.
- BM25
- : A classic keyword scoring method that ranks documents by matching terms.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- reciprocal rank fusion
- : A way to merge two ranked result lists by adding 1 / (k + rank) for each list.
- reranker
- : A second model that rescores a short list of search results by reading query and passage together.
- cross-encoder
- : A model that reads a query and a passage together and scores how relevant the passage is. Used for reranking.
- chunk
- : A small piece of a document, sized for indexing and retrieval.
- agent
- : A model run in a loop that chooses its own tools and steps until the task is done.
- GraphRAG
- : Retrieval that first builds a graph of entities and relations, for questions that span many documents.
- recall
- : Of the truly positive cases, the share the system found.
- latency
- : The time you wait for a response.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
Techniques
: combine keyword search (, which scores documents by matching term frequency) with search, merged with . : fetch 50 candidates cheaply, then score each against the query with a stronger model that reads query and passage together, and keep the top 5. Contextual : prepend a sentence of context to each chunk before indexing, so "revenue grew 3 percent" knows which company and quarter it belongs to. Query rewriting: turn a vague question into several search queries. : let the model decide what to search for, and search again if results are weak. : build a graph of entities and relations for questions that span documents.
Measure each one
Create a set of questions with known source passages. Track at k, meaning how often the right passage is in the top k results. Add a technique, rerun, and keep it only if the number improves enough to justify the extra and cost.
A worked example: reranking
Search fetches 50 candidates quickly. A reranker reads each candidate together with the question and scores relevance more carefully, and the top 5 go to the model. The first stage is cheap and broad, the second is slower and sharper. Think of a recruiter screening 50 resumes by keywords, then interviewing the best five.
When to consider graph approaches
Questions like "what are the main themes across all these 500 reports?" need a view of the whole collection, not just the nearest passages. GraphRAG builds a map of entities and relations first. It costs more to build. Use it only when plain retrieval cannot answer such questions.
Hybrid search: merging two lists
Keyword scores and vector scores use different scales, so you cannot add them. Reciprocal rank fusion (RRF) uses only the rank. Each list gives a document 1 / (60 + rank), and the two scores are added. The number 60 is a common choice.
. Document A is rank 1 in keywords and rank 3 in vectors: 1/61 + 1/63 = 0.0164 + 0.0159 = 0.0323. Document B is rank 2 in keywords and rank 1 in vectors: 1/62 + 1/61 = 0.0161 + 0.0164 = 0.0325. Document C is rank 1 in keywords only: 0.0164. B wins narrowly, then A, then C. A document that appears in both lists beats one that is first in only one.
What goes wrong: the reranker cannot find what was never fetched
A reranker only reorders the 50 candidates it is given. If the right passage is not among them, a better reranker cannot help. The symptom is that answers stay wrong after you add reranking. Confirm it by measuring recall at 50 before the reranker. If the right passage is missing from the top 50, fix the first stage with better chunks or hybrid search. If it is there but not in the top 5, the reranker is the place to work.
When simple retrieval is enough
With 200 short pages, plain vector search may already put the right passage in the top 5 for 95 of 100 test questions. Reranking would add a model call over 50 pairs for the last few points. Run the numbers first. Stop adding techniques when the remaining misses are things retrieval cannot fix, such as a question the documents do not answer.
Common mistake
Adding techniques because they are popular, not because a failing metric called for them.
A basic RAG bot answers 70 percent of questions well. The team adds three things in turn: a keyword search alongside the vector search, a reranker, and a one-sentence context note on each chunk. Recall goes from 0.70 to 0.78 to 0.85 to 0.91. Each step was small. All were measured.
Check yourself
What does a reranker do?
Not quite. It is slower but more accurate, so it runs on a short list.
How do you decide to keep a new technique?
Not quite. Measure on your own questions.
GraphRAG builds a graph of entities first. When might it help most?
Not quite. Collection-wide questions need a map of the whole set, not just the nearest passages.
You got 0 of 3 right.
Go deeper
- Article
Anthropic · Measured gains from contextual chunks, BM25 and reranking.
- Repo
Nir Diamant · Notebooks for many techniques.
- Paper
Edge et al. · Microsoft's GraphRAG paper.
Spotted a mistake or something unclear? Report it on GitHub.