RAG or fine-tuning?

The differenceCopy Anchor Link

RAG puts the right facts in the prompt at question time. The model is unchanged. Fine-tuning updates the model's weights on examples so it adopts a style, format or narrow skill. Fine-tuning is weak at adding facts, hard to update and hard to audit. RAG is easy to update and to cite.

A decision guideCopy Anchor Link

Does the model need facts that change or are private? Use RAG. Does it need a consistent tone, a fixed output format or to follow a complicated procedure reliably? Try prompts and examples, then consider fine-tuning. Do you need a small cheap model to match a big one on a narrow task? Fine-tune it, using the big model's outputs as training data. Many production systems use both: a fine-tuned model for style and format, and RAG for facts.

What each one costs youCopy Anchor Link

RAG adds work on every call: a search, extra prompt text and sometimes a reranker. Its setup is light, and updating knowledge means re-indexing a document. Fine-tuning has the opposite shape. It needs a set of good example inputs and outputs that you write or collect, a training run and a test that the model did not get worse elsewhere. Once trained, calls are not slower, and the prompt can be shorter.

Changing a fact is the clearest difference. With RAG you edit the document. With fine-tuning you must prepare new examples and train again, and you cannot easily point to where a fact lives in the weights.

A second example: neither is neededCopy Anchor Link

A team wants replies in a fixed three-line format. They try a clear instruction plus three example replies in the prompt, and the format comes out right in 19 of 20 tests (invented numbers). That costs nothing to maintain. Fine-tuning would add training and testing for a gain they do not need. Always try the prompt first.

What goes wrongCopy Anchor Link

A fine-tuned model states old facts with full confidence, because they are baked into its weights. To check, ask it a question whose answer changed after training. A RAG system fails differently: the right chunk is not found, so the model guesses or declines. Look at the retrieved chunks to tell which one you have.

Fine-tuning on a small, uneven set can also narrow the model. It copies the style of the examples and gets worse on other tasks, which is why you test on questions outside the training set.

Try it yourselfCopy Anchor Link

Pick one task, such as answering questions about a small policy document. Try three setups on the same 10 questions: the prompt alone, the prompt with the document pasted in, and (if the document is too long to paste) the prompt with retrieved excerpts. Score each by counting correct answers. Expect the first to guess, and the others to improve sharply. That gap is what RAG buys, with no training at all.

Common mistakeCopy Anchor Link

Reaching for fine-tuning first because it sounds advanced. It is usually the more expensive, slower, harder-to-update option.