Regression testing prompts and models
Every change to a prompt, model, tool or retrieval setting can break something that worked. A fixed test set catches that before users do.
Words on this page (7)
- prompt
- : The input text you give a model, including instructions, examples and data.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- retrieval
- : Finding the most relevant pieces of text for a query.
- CI
- : Continuous integration: automatically running tests on every code change. CD deploys passing changes.
- baseline
- : A simple method, such as "same as last week", that any model must beat to be worth its cost.
- temperature
- : A setting that sharpens or flattens the probabilities before sampling. Low is predictable, high is varied.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
The loop
Keep a versioned set of cases with expected behavior. Run it on every change in . Compare against the last accepted and block merges that lower the pass rate beyond a threshold you chose. Store the outputs, so a failing case can be opened and read, and a human can accept an intentional change by updating the baseline.
Handling variability
Outputs vary, so a single failure may be noise. Run each case several times and compare pass rates, or fix and seeds where possible. Use deterministic assertions where you can and a calibrated judge where you cannot.
Model upgrades
A new model version is a code change. Run the full suite on it before switching. Providers retire models, so keep a tested fallback. After switching, watch production metrics, since your test set will never cover everything.
A worked example: a CI gate
Test set: 60 cases. Baseline pass rate: 91 percent. A change drops it to 86 percent. The pipeline fails the merge and lists the five newly failing cases. The author looks, finds that a new instruction confused the date format, fixes it, and the rate returns to 92 percent. The cost: a few minutes and some .
Upgrading models
Treat a new model version like a library upgrade: run the full suite first, compare cost and speed, keep the old one as a fallback until you are confident.
Setting the threshold
Run the unchanged system five times on the 60 cases. Invented results: 54, 55, 53, 55 and 54 passes. The spread is two cases, or 3.3 points, with no change at all. A threshold tighter than that blocks good merges at random. Set it above the natural spread, for example 5 points. Five runs underestimate the real spread, so leave some margin.
The total can hide swaps
Pass counts can match while the behavior changed. A new fixes 3 old failures and breaks 3 cases that used to pass. The total stays at 55, and the gate stays green.
So compare case by case, not only the totals. List the cases that went from pass to fail. Those are the regressions, even when other cases improved.
What goes wrong: the flaky gate
Symptom: the build fails, someone reruns it, it passes, and nobody looks at the failure any more. To confirm, run the same commit five times. If the result changes, the gate is noisy. Run each case three times and take the majority, or replace the noisy checks with exact ones. A gate people learn to ignore is worse than none.
Common mistake
Updating baselines casually. Only accept a lower score when a person has decided the trade is worth it.
A company upgrades to a newer model because it is advertised as better. Two days later, a prompt that extracted dates correctly starts returning them in a different format, and a downstream report breaks. A saved test set would have flagged it in the pull request, before any customer saw it.
Check yourself
What is a good rule for merging a prompt change?
Not quite. A fixed threshold makes the decision objective.
Why run each case several times?
Not quite. Compare pass rates, not single outcomes.
A provider retires your model version next month. What is the safe approach?
Not quite. Treat a model version change like a library upgrade and test it first.
You got 0 of 3 right.
Go deeper
- Repo
promptfoo · Runs test suites in CI and compares prompts and models.
- Article
Hamel Husain · Build the test set from real failures.
- Repo
OpenAI · A framework and registry of tests.
Spotted a mistake or something unclear? Report it on GitHub.