Pretraining, fine-tuning and RL

What it isCopy Anchor Link

A chatbot is built in stages. Each stage teaches it something different.

Stage one is pretraining. The model reads a huge amount of text and learns to predict the next token. This is where its knowledge and its ability with language come from. The result is a base model. It can continue any text, but it does not know it is supposed to help you.

Stage two is supervised fine-tuning, or SFT. People write thousands of example conversations: a question, then a good answer. The model trains on them and starts to answer like an assistant.

Stage three is preference tuning. People compare two answers and pick the better one, and the model is nudged toward the kind people prefer. You will see the names RLHF and DPO for this. Both mean learning from human preferences.

Stage four, used for reasoning models, is reinforcement learning on tasks with checkable answers, such as math and code. The model gets a reward when the final answer is right, so it learns to think longer and more carefully.

Why it mattersCopy Anchor Link

Pretraining has a cutoff date, so the model knows nothing newer. Politeness, refusals and tool use come from the later stages. That is why a prompt can change how a model behaves, but cannot reliably teach it new facts.

When you need new facts, retrieve them and put them in the prompt. When you need the same style or format every time at large scale, consider fine-tuning.

A worked example: what each stage addsCopy Anchor Link

Ask a base model "How do I bake bread?" It might continue with "How do I bake a cake? How do I bake cookies?", like a list of search suggestions. After instruction tuning it gives steps. After preference tuning the steps get clearer, safer and better organized. After reasoning training it can work through a long problem, such as scaling a recipe for 37 people.

What one training example looks likeCopy Anchor Link

In pretraining, one sentence produces many practice questions. Take the six tokens "The cat sat on the mat" (invented split). The model is shown "The" and must predict "cat". Then "The cat" and must predict "sat". That gives 5 predictions from one sentence, each scored by how much probability the right token got. Billions of such sentences make up the run.

Fine-tuning uses the same scoring, but on conversations, and usually only the assistant's reply counts. Same mechanism, different data.

Trade-offs: prompt, retrieve or fine-tuneCopy Anchor Link

Match the tool to what is missing.

Missing facts that change often: retrieve them. A company with 200 policy pages updated weekly cannot re-tune each week, and retrieval can show its sources.

Missing behavior you can describe: write a clearer prompt first, since it is free and instant.

Missing consistency at huge scale: fine-tune a small model. A brand voice in a fixed three line format, millions of times a day, is a good fit. The costs are collecting good examples and the risk of side effects.

What goes wrong after tuningCopy Anchor Link

Symptoms to watch for:

1. A tuned model gets worse at unrelated tasks, because the update shifted weights it needed.
2. Answers have the right format but wrong facts. Format can be taught. Knowledge mostly cannot.
3. Reinforcement learning on checkable tasks can be gamed. A coding model might special-case the exact tests instead of solving the problem.

To confirm, test on examples that were not used in tuning. Also rerun a small set of general tasks before and after the change.

Common mistakeCopy Anchor Link

Fine-tuning a model to "teach" it your company facts. Facts are better supplied when the question is asked, through retrieval. That keeps them up to date and shows the sources.

The math, when you are readyCopy Anchor Link

Training is repeated guessing, measuring the error and nudging the weights. The numbers behind that loop are in Loss and gradient descent. It is optional for now.