Retrieval-augmented generation
Retrieve relevant text first, then ask the model to answer using it. It is how you give a model knowledge it was not trained on.
Retrieval-augmented generation, step by step
Question: What is our refund window?
vector: [0.12, -0.40, 0.88, 0.05, …]
- 0.82Refunds are accepted within 30 days of purchase.
- 0.31Shipping takes 3 to 5 business days.
- 0.64Gift cards cannot be refunded.
- 0.12Our office is in Berlin.
- 0.71Refund requests are processed within 5 business days.
Step 1 of 6
A user asks a question. The system will fetch relevant text first, then ask the model to answer from it.
Step 2 of 6
Embed the question: an embedding model turns it into a vector, the same kind used to index the chunks earlier.
Step 3 of 6
Search: compare the question vector with every chunk vector. Each chunk gets a similarity score. This is the retrieval step.
Step 4 of 6
Keep the best few. A reranker can re-order them by reading each one with the question. Here the top three stay.
Step 5 of 6
Build the prompt: instructions, the retrieved chunks and the question. The model sees only this, so whatever retrieval missed is invisible to it.
Step 6 of 6
The model answers from the chunks and cites them. If the chunks had lacked the answer, the prompt tells it to say so.
Words on this page (12)
- chunk
- : A small piece of a document, sized for indexing and retrieval.
- metadata
- : Extra labels stored with a document or chunk, such as its title, date or country, which let you filter results.
- retrieval
- : Finding the most relevant pieces of text for a query.
- prompt
- : The input text you give a model, including instructions, examples and data.
- RAG
- : Retrieval-augmented generation: look up relevant text first, then give it to the model to answer from.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- policy
- : A rule for choosing actions. In a language model, the model itself is the policy.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- hybrid search
- : Combining keyword search and vector search.
- reranker
- : A second model that rescores a short list of search results by reading query and passage together.
- eval
- : A repeatable test of AI output quality, run before and after every change.
- prompt injection
- : Untrusted text that contains instructions the model follows, such as a web page telling it to leak data.
The three parts
Indexing (preparing the documents ahead of time): split them into , embed each one and store it with , which means extra labels such as title or date. : embed the question, fetch the closest chunks, optionally filter (keep only chunks with the right label, such as a year) and rerank them. Generation: put the chunks in the with instructions to answer only from them and to cite which chunk each claim came from.
Why use it
It adds private or fresh knowledge without retraining, lets you cite sources, lets you update by re-indexing, and keeps access control possible because you decide which chunks a user can see. It does not make a model reason better. It gives it better material.
Try it
The demo above steps a question through indexing, retrieval and generation.
A worked example: one question end to end
Index time: the handbook is split into 400 chunks, each embedded and stored with its title. Query time: the question is embedded, the five nearest chunks are fetched, a short prompt says "Answer only from these excerpts and cite them. If they do not contain the answer, say so." The model replies with the answer and a citation. A person can click the citation and verify.
What the model actually receives
The model never sees your database. It sees one long prompt that your code assembled. In order, it holds the rules (answer only from the excerpts, cite them, admit gaps), then each retrieved chunk with a label such as its title and section, then the user's question. The model reads it like any other text. Because of this, a answer can only be as good as what your code pasted in.
A second example: when RAG is the wrong tool
"Summarize this 3-page contract" needs no retrieval, because the whole text fits in the prompt. Retrieval would only risk leaving out a clause. A question like "How many of our policies mention data retention?" is also a poor fit. Retrieval returns the five best-matching chunks, not every match, so the count would be wrong. Questions that need the whole collection need a different , such as a database query (a precise lookup written in a language like SQL).
What goes wrong
Failures you will see:
- A confident wrong answer, because the right chunk was missing and the prompt did not allow "I do not know".
- A blended answer, because two chunks disagree (a 2019 and a 2024 one) and the model mixed them.
- A citation that does not support the claim. The model attached a real source to a sentence it made up.
- An outdated answer, because the index was not rebuilt after the document changed.
For each, look at the retrieved chunks first ( them in your code). Then ask two questions in order: was the right chunk there, and did the answer use it? The first decides whether to fix retrieval. The second decides whether to fix the prompt.
Most bad answers turn out to be retrieval failures: the right chunk was not retrieved, was split badly or ranked below noise. Common fixes are better chunking, short notes on each chunk saying where it came from (contextual notes), combining keyword and meaning search () and a .
Try it yourself
Paste five short paragraphs from any handbook into a chat with a model. Ask three questions the paragraphs answer and two they do not. Add the instruction to answer only from the excerpts and admit gaps. Expect correct, cited answers to the first three and "the excerpts do not say" to the last two. Remove the instruction and ask the two unanswerable questions again. Compare how often the model guesses instead.
Common mistakes
Skipping the "say you do not know" instruction. Letting retrieval pull documents a user is not allowed to see.
Where this connects
You will need a way to tell whether answers are getting better. Evaluating RAG is the next lessons here, and Why evals come first explains the habit. Retrieved text can also contain instructions: see Prompt injection and the lethal trifecta.
A new hire asks the company bot: "How many vacation days do I get after one year?" A plain model guesses a typical number. A RAG bot first searches the HR handbook, finds "Employees receive 25 days after 12 months", pastes that paragraph into the prompt and replies "25 days, per the Time Off policy, section 3." The model did not become smarter. It was given the right page.
Check yourself
A RAG answer is wrong. What do you check first?
Not quite. Retrieval quality is the most common root cause, and it can be tested without the model.
What should the prompt tell the model to do when the sources lack the answer?
Not quite. Allowing "I do not know" cuts confident wrong answers.
The right chunk was retrieved but the answer ignores it. What should you adjust?
Not quite. Retrieval did its job, so the instruction or the model is the weak link.
You got 0 of 3 right.
Go deeper
- Course
Nir Diamant · Notebooks for dozens of RAG methods with explanations.
- Article
Anthropic · Prepending chunk context and combining BM25 with embeddings.
- Paper
Lewis et al. · The paper that named RAG.
- Article
Eugene Yan · RAG, evals, guardrails and feedback as design patterns.
- Repo
InfiniFlow · An open-source RAG engine with document parsing.
Spotted a mistake or something unclear? Report it on GitHub.