Limits and hallucinations
Models produce plausible text, not verified facts. Knowing the failure modes tells you where to add checks.
Words on this page (8)
- hallucination
- : A confident answer that is not supported by the model's knowledge or the provided sources.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- knowledge cutoff
- : The date after which the model has seen no training data.
- schema
- : A description of the shape of data: which fields exist and what types they have.
- prompt
- : The input text you give a model, including instructions, examples and data.
- eval
- : A repeatable test of AI output quality, run before and after every change.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
What it is
A is a confident answer that is not supported by the model's knowledge or the provided sources. It happens because the objective rewards fluent continuations, and the model has no built-in signal for "I do not know". Common failure areas are precise facts and citations, arithmetic done in its head, events after the , and long tasks where an early mistake carries forward.
Capabilities are also uneven. A model can solve a hard proof and then miscount letters in a word.
Why it matters
Design around the failure instead of hoping it disappears. Give the model sources and ask for quotes. Let it call a calculator or run code instead of computing in its head. Add verification steps that check the output against something independent, such as tests, a or a second look from a separate . Then measure how often it still fails (see the chapter).
Worked example: three guard rails
Task: summarize a customer's contract. Rail one: give the model the contract text and tell it to answer only from it. Rail two: require a short quote for every claim, which you can check with a text search. Rail three: have a second prompt (or a person) verify a sample. None of these make the model perfect. Together they make errors visible.
Where models are weak
Exact arithmetic with long numbers, precise dates and names that are rare, anything after the knowledge cutoff, and counting letters in words ( hide letters). Where possible give the model a : a calculator, a search, a code runner.
Why a model makes things up
Think about what happens with a rare fact. Ask for the author of an obscure book. The model has seen little about it, so its probabilities spread across many plausible names. Invented numbers: 12 percent for one name, 10 percent for another, 9 percent for a third. The sampler must still pick something, and the top choice at 12 percent is printed in the same confident tone as a name with 95 percent behind it.
Nothing in that loop separates "I know" from "this looks right". Bigger models tend to make fewer such errors, but none are immune.
A second example: matching checks to stakes
Guard rails cost time and money, so scale them to what a mistake costs. Rewriting an email to sound politer has few facts to get wrong, and a quick read by you is enough. Summarizing a legal contract needs the full set of rails.
Even a good rail can mislead. A required quote might exist in the contract but not support the claim it is attached to. Checking that the quote exists is not the same as checking that it says what the model claims.
How to tell it is working
Build a small test set of about 30 questions with known answers. Include some whose answers are not in the provided document. Count three things: right answers, wrong answers, and times the model said the answer is not available.
A well-grounded setup should say "not stated" when asked, for example, the notice period of a contract that never mentions one. If it invents a number, your rails are not yet working. Rerun the set after every change to the prompt or model so you see whether the wrong-answer rate moved.
Common mistake
Reading confidence as accuracy. The tone is equally fluent when the answer is wrong.
A lawyer once submitted a legal brief containing six court cases that a chatbot had invented, with convincing names and citations. None existed. The model was not lying. It produced text that looked like legal citations, because that is what it is built to do. The lawyer skipped the step of checking.
Check yourself
Which fix reduces arithmetic mistakes most reliably?
Not quite. Hand exact computation to a tool instead of asking the model to compute it token by token.
How can you reduce made-up citations?
Not quite. Grounding in given text and requiring checkable quotes gives you something to verify mechanically.
The model states a date in a confident tone. What does the tone tell you?
Not quite. Fluent confidence is the same whether the answer is right or wrong, so check what matters.
You got 0 of 3 right.
Go deeper
- Article
OpenAI · Explains how training and benchmarks reward guessing over abstaining.
- Article
Yan, Husain, Liu and others · Practitioner lessons on prompts, retrieval, evals and operations.
- Article
Chip Huyen · Where LLM apps break and how teams handle it.
Spotted a mistake or something unclear? Report it on GitHub.