Transformers and attention
The transformer lets every token look at every earlier token and decide which ones matter for the next prediction.
Words on this page (13)
- transformer
- : The neural network architecture behind modern language models. It uses attention to relate every token to the others.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- embedding
- : A list of numbers that represents the meaning of a piece of text or an image, so similar things sit close together.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- prompt
- : The input text you give a model, including instructions, examples and data.
- KV cache
- : Stored keys and values of earlier tokens, reused during generation so each new token is cheap to compute.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- tokenizer
- : The program that splits text into tokens and maps each one to an integer id.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- training
- : Adjusting a model's parameters on data so its predictions improve.
What it is
The is the design behind modern language models. Its key idea is that every can look at every earlier token and decide which ones matter.
Here is the path a piece of text takes. First it is split into tokens. Each token becomes a , which is a list of numbers called an . Then the vectors pass through a stack of identical blocks. Each block does two jobs. lets a token pull in information from other tokens. A feed-forward step then thinks about each token on its own. At the very end, the last vector is turned into a score for every possible next token.
In attention, each token makes three vectors. The query says what it is looking for. The key says what it contains. The value says what it offers. A token compares its query with every earlier key, and the best matches contribute their values. This is how the word "it" can pull in the meaning of the noun it refers to.
Why it matters
You do not need the math to use models, but this picture explains real limits. Attention compares every token with every earlier token, so the work grows quickly with length. That is why long inputs cost more. It also explains why position in a matters, and why models can copy a pattern from examples you give them. Earlier keys and values are saved in a , so repeated prefixes are cheap. That idea is behind prompt caching.
A worked example
Take "The bank of the river was muddy." When the model handles "bank", attention pulls in "river", so the vector for "bank" moves toward "edge of water". In "The bank approved my loan", it pulls in "loan" and "approved" and moves toward "financial institution". The word is the same, but its meaning is set by its neighbors.
Go deeper
Watch the 3Blue1Brown chapters first. Then play with an interactive visualization. Then read the original paper or the illustrated walkthrough.
A trace with small numbers
Invented numbers for illustration. A token has three earlier tokens to look at. Its query compared with their keys gives scores 3, 1 and 0. turns these into : e^3 is about 20.1, e^1 about 2.7, e^0 equals 1, total about 23.8. The weights are about 0.84, 0.11 and 0.04.
Say the three values are single numbers, 10, 20 and 30. The output is the weighted average: 0.84 × 10 + 0.11 × 20 + 0.04 × 30, about 12.0. Most of the answer comes from the first token, because its key matched the query best. Real models scale the scores before softmax and use long vectors, but the steps are the same.
Trade-offs and limits
Each comparison costs work, and the number of comparisons grows with the square of the input length. As the history chapter showed, doubling the input quadruples this part of the cost.
A model also runs several attention heads in parallel. A head is one independent copy of attention that can focus on a different kind of relationship. Attention weights show where information flowed, but they are only part of the computation, so they are an incomplete explanation of why the model answered as it did.
A second example: where attention does not help
Attention can only mix information the vectors already hold. Ask a model how many r letters are in "strawberry". The may have hidden the letters inside larger pieces, so the model has little direct access to the letters. Giving the model a code avoids the problem.
A different failure is a fact buried in a very long prompt. Weights spread thin across many tokens, and the model may ignore a line that is present. To confirm, move the fact to the start or end of the prompt and ask again. If the answer changes, position was the cause.
Common mistake
Thinking more layers always means a smarter model. Depth helps, but data and matter just as much.
The math, when you are ready
Attention is a few dot products, a softmax and a weighted average. The step-by-step numbers are in Attention in numbers, in the Building Blocks chapter later in the path. Read it after you have used and built with models, and it will feel much easier.
A student reads a textbook paragraph and, before answering a question, flips back through the pages looking for the lines that matter. A transformer does this for every word it writes: it looks back over everything so far, scores what is relevant, and uses that. The ability to look back, in parallel, is why models can follow a long instruction or keep a story consistent.
Check yourself
What does attention let a token do?
Not quite. Queries are matched against keys of earlier tokens, and the best matches contribute their values.
Why are very long inputs expensive?
Not quite. Cost rises roughly with the square of the length because of those comparisons.
In "The bank approved my loan", what moves the meaning of "bank" toward money?
Not quite. Neighbouring words set the meaning: attention blends their information into the word's vector.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · The clearest animated explanation of embeddings and attention.
- Tool
Georgia Tech Polo Club · Runs GPT-2 in your browser and lets you change temperature live.
- Tool
Brendan Bycroft · A 3D walkthrough of every operation in a small GPT.
- ArticleThe Illustrated TransformerArticle
Jay Alammar · The classic diagram-by-diagram walkthrough.
- Paper
Vaswani et al. · The original transformer paper.
Spotted a mistake or something unclear? Report it on GitHub.