Fine-tuning with LoRA and QLoRA
Fine-tuning adjusts a pretrained model on your examples. Use it to change behavior and style, not to add facts.
Words on this page (13)
- fine-tuning
- : Further training on a smaller, targeted set of examples to change behavior or style.
- prompt
- : The input text you give a model, including instructions, examples and data.
- retrieval
- : Finding the most relevant pieces of text for a query.
- LoRA
- : A cheap fine-tuning method that freezes the model and learns small add-on matrices. QLoRA also stores the base in 4-bit.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- matrix
- : A grid of numbers. Multiplying a vector by a matrix transforms it. Model layers are mostly matrices.
- GPU
- : Graphics processing unit. A chip with thousands of cores, ideal for the matrix math in AI.
- eval
- : A repeatable test of AI output quality, run before and after every change.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- policy
- : A rule for choosing actions. In a language model, the model itself is the policy.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- overfitting
- : When a model memorizes training data and does worse on new data.
When to fine-tune
First try better , examples in context and . when you need a consistent format or tone at scale, want a small cheap model to match a big one on a narrow task, or have prompt text that is too long to send every call. Do not fine-tune to teach new facts. It is unreliable for that and retrieval is easier to update.
LoRA and QLoRA
Full fine-tuning updates every , which needs a lot of memory. (low-rank adaptation) freezes the original weights and learns two small per layer whose product is the update. The worked example below puts numbers on how small that update is. QLoRA also stores the frozen base in 4-bit form so a large model fits on one .
Data is the work
Quality beats quantity. A few hundred to a few thousand clean examples often suffice. Hold out an set, compare against the base model with prompting, and check that you did not make other abilities worse.
A worked example: LoRA in numbers
A weight matrix of 4,096 by 4,096 holds 16.8 million numbers. Full fine-tuning would update all of them, for every layer. LoRA freezes them and learns two thin matrices, 4,096 by 8 and 8 by 4,096, about 32,800 numbers each, 65,500 in total, about 0.4 percent of the original. The update is their product, added to the original. Far less memory, and the add-on file is small enough to share.
How LoRA changes a layer
For an input x of 4,096 numbers, the layer computes the frozen output plus a small correction. The correction multiplies x by A, which squeezes it down to 8 numbers, then by B, which expands those 8 back to 4,096. Because 8 numbers carry the whole change, the update can only move the layer along 8 directions. That limit is called the rank. B starts as all zeros, so before the correction is zero and the model behaves exactly like the original. After training, you can add B A into W once, so serving costs no extra time.
A second example: when not to fine-tune
A shop wants a bot that knows its return , which changes every month. Fine-tuning would mean building new training examples and retraining each time, and the model might still quote last month's rule. Putting the policy in a document that a retrieval step fetches means editing one file. Fine-tune for how the bot answers. Retrieve for what it knows.
What goes wrong: gains that hide losses
After fine-tuning on a narrow set, the format may be perfect while the model gets worse at everything else. It may refuse normal requests, or answer unrelated questions in the style of your examples. To confirm, keep a fixed set of about 20 general prompts. Run them on the base model and on the tuned one, and read both sets of answers side by side. Also watch validation on held-out examples: if it rises while training loss falls, you are . Fewer training passes or a smaller rank usually helps.
Common mistake
Fine-tuning before trying prompting, examples and retrieval.
A law firm wants every contract summary in exactly their house format: three headings, no more than 150 words, fixed phrases. A long prompt works 90 percent of the time. After fine-tuning a small model on 800 of their best summaries, it follows the format 99 percent of the time, faster and cheaper. Fine-tuning shines when you need the same behavior reliably, at scale.
Check yourself
You want a model to know last week's product changes. Should you fine-tune?
Not quite. Fine-tuning is poor at adding facts and hard to update.
A rank-8 LoRA on a 4096 by 4096 matrix stores about what share of its numbers?
Not quite. Two 4096 by 8 matrices are 65,536 numbers in total, versus 16.8 million: about 0.4 percent.
You have 800 clean examples and a prompt that works 90 percent of the time. What should you try first?
Not quite. Prompting is cheaper to try. Fine-tune when it falls short on your tests.
You got 0 of 3 right.
Go deeper
- Paper
Hu et al. · The LoRA paper.
- Paper
Dettmers et al. · Fine-tuning large models on a single GPU.
- Repo
Unsloth · Fast, memory-efficient fine-tuning with notebooks.
- Repo
hiyouga · A unified tool for fine-tuning many models.
- Course
Hugging Face · A practical course on aligning small models.
- DocsPEFTDocs
Hugging Face · The standard library for LoRA and related methods.
Spotted a mistake or something unclear? Report it on GitHub.