The prompt engineering era
What survived from prompt engineering is a handful of techniques with a mechanism behind them. The magic-words era mostly did not.
Words on this page (11)
- prompt
- : The input text you give a model, including instructions, examples and data.
- few-shot
- : Giving a few worked examples in the prompt.
- in-context learning
- : A model picking up a task from examples inside the prompt, without any retraining.
- chain of thought
- : Having the model write out intermediate reasoning steps before the final answer.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- tool use
- : The model asks your program to run a function and reads the result.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- agent
- : A model run in a loop that chooses its own tools and steps until the task is done.
- skill
- : A folder of instructions and scripts an agent loads only when a task needs it.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
- training
- : Adjusting a model's parameters on data so its predictions improve.
What was real
examples work because the model infers the task from patterns in context, which is . works because written reasoning gives later something to condition on. Clear structure and delimiters help because the model has to tell instructions from data. ReAct interleaved reasoning with and is the direct ancestor of loops.
What was hype
Secret phrases that supposedly unlock hidden performance, "act as a world-class expert", and marketplaces. Much of this faded as models got better at following plain instructions. "Prompt engineer" as a standalone job title mostly dissolved into general software and product work, because a prompt is a small part of a system that also needs data, and tests.
What it became
Prompts are now treated as code: versioned, tested against examples and reviewed. Writing one is still a . It is just no longer the whole job.
A worked example: separating habit from superstition
"You are a world-class expert": no measurable effect on many tasks. "Think step by step": real effect on multi-step problems for older models. "Here are three examples": real, reliable effect on format. "Take a deep breath and work on this problem step-by-step": a 2023 DeepMind paper found it reached 80.2 percent on a grade-school math set with one model (PaLM 2-L), and I found no evidence of a reliable effect on current models. The test is simple: run both versions on twenty inputs and count.
What became of the job title
Writing prompts turned into one of many skills. Today it is usually bundled with building the whole system: data, tools, tests and monitoring.
How few-shot works, with a tiny trace
Invented prompt: "Review: great -> positive. Review: awful -> negative. Review: fine ->". The model has seen no on this exact list. It reads the pattern and continues it, so the next word is likely "positive" or "neutral" in the same one-word style.
Without the two examples, the same model might answer with a full sentence. The examples teach the format as well as the task. To measure it, run twenty inputs both ways and count outputs in the right format. Invented result: 11 of 20 without examples, 19 of 20 with.
What goes wrong: tuning to your test cases
You rewrite a prompt until three test inputs all pass. On twenty new inputs it fails half of them. The prompt learned your three cases, much like a student who memorizes the practice exam.
To confirm, split your inputs into two groups. Tune the prompt on the first group only, then score the second group once, without changing anything. If the second score is far lower, you overfit the prompt. Keep the second group hidden until the end.
When a technique is not needed
Chain of thought helps on multi-step problems, such as a bill with a discount and tax. It adds cost to a one-step lookup like "What is the capital of France?", because the model writes extra tokens it does not need. Reasoning models are trained to think before they answer. Adding "think step by step" to one may change nothing. Run both versions on your task before you keep the phrase.
Common mistake
Copying prompts from a social media thread without testing them on your own task.
In 2023 people shared "magic prompts" like secret codes. Some were just good habits (be specific, give examples). Others only looked clever. As models got better at plain instructions, the secret-code ones faded and the habits stayed. What survived was the part that had a reason behind it.
Check yourself
Why does few-shot prompting work?
Not quite. In-context learning lets the model pick up the task from examples in the prompt.
What happened to the "prompt engineer" job title?
Not quite. A prompt is one part of a system that also needs data, tools and tests.
You find a viral "magic prompt" online. How do you decide whether to use it?
Not quite. Count results on twenty inputs before adopting any prompt.
You got 0 of 3 right.
Go deeper
- Paper
Brown et al. · The origin of few-shot prompting.
- Paper
Wei et al. · Step-by-step reasoning improves accuracy.
- Paper
Yao et al. · Reasoning and tool use in one loop.
- ArticlePrompt EngineeringArticle
Lilian Weng · A clear survey of techniques from the peak of the era.
Spotted a mistake or something unclear? Report it on GitHub.