Attention in numbers
Attention is dot products, a softmax and a weighted average. Here it is with a two-token example.
Attention for one query: turn it toward a key
weight 67%
weight 33%
output = 0.67 × [10, 0] + 0.33 × [0, 10] = [6.7, 3.3]
At 0° this is the lesson's example. The output leans toward whichever key the query matches best.
Words on this page (11)
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- dot product
- : Multiply matching entries of two vectors and add them up. Large when they point the same way.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- matrix
- : A grid of numbers. Multiplying a vector by a matrix transforms it. Model layers are mostly matrices.
- gradient
- : The direction and rate at which the loss changes when each parameter changes. Training steps against it.
- KV cache
- : Stored keys and values of earlier tokens, reused during generation so each new token is cheap to compute.
- prompt
- : The input text you give a model, including instructions, examples and data.
- training
- : Adjusting a model's parameters on data so its predictions improve.
The formula
Each has a query q, a key k and a value v, all . For one query, the score against each key is a scaled by the square root of the key size d: score = q · k / √d. turns the scores into . The output is the weighted sum of the values: out = Σ weight × v. In form: softmax(Q Kᵀ / √d) V.
Example
Let d = 2, with q = [1, 0]. Two keys: k₁ = [1, 0] and k₂ = [0, 1]. Values: v₁ = [10, 0], v₂ = [0, 10]. Scores: q·k₁ = 1, q·k₂ = 0. Scaled by √2 = 1.414: 0.71 and 0. Softmax: e^0.71 = 2.03, e⁰ = 1, sum 3.03, so weights 0.67 and 0.33. Output = 0.67 × [10, 0] + 0.33 × [0, 10] = [6.7, 3.3]. The query matched the first key more, so the output leans toward v₁.
Why divide by √d
Dot products grow with vector size, which would push softmax into near one-hot outputs with tiny . Scaling keeps the scores in a useful range.
In a real model
A causal mask blocks each token from seeing later tokens. Many heads run in parallel, each learning a different pattern, and their outputs are concatenated. The key and value vectors of earlier tokens are cached during generation, which is the behind fast decoding and caching.
In plain words
Each word makes three vectors. The query says "what am I looking for?". The key says "what do I contain?". The value says "what do I give if you pick me?". A word compares its query with everyone's keys, turns the scores into percentages with softmax, and takes a weighted mix of the values.
A second example with a causal mask
Take three tokens and suppose token 2 scores [1, 2, 3] against tokens 1, 2 and 3. Without a mask, softmax gives e¹ = 2.718, e² = 7.389, e³ = 20.086, total 30.193, so weights 0.090, 0.245 and 0.665. But token 2 must not see token 3, which comes later. The mask removes that score before softmax, so only [1, 2] remain: 2.718 / 10.107 = 0.269 and 7.389 / 10.107 = 0.731. Token 3 gets weight 0. Without the mask, a model could cheat on next-word prediction by reading the answer.
Why the scaling matters, in numbers
Suppose each entry of q and k is a random number with spread about 1. A dot product adds d such products, so its typical size grows like √d. With d = 64 that is about 8. Unscaled scores of [8, -8] give weights 1 / (1 + e⁻¹⁶) = 0.9999999. The softmax is nearly saturated: its slope, p × (1 - p), is about 0.0000001, so learns almost nothing. Divide by √64 = 8 and the scores become [1, -1]. The weights are 0.881 and 0.119, and the slope is 0.105, about a million times larger.
Trade-offs: cost grows with the square
Every token scores against every earlier token, so n tokens give about n² scores per head. At 1,000 tokens that is 1,000,000. At 2,000 tokens it is 4,000,000: double the text, four times the scores. This is why long prompts get slower and costly, and why much research targets cheaper attention. The KV cache avoids recomputing old keys and values, but it grows linearly with length and takes memory.
Check it by hand
Reuse the lesson's keys and values but change the query to q = [0, 1]. Scores: 0 and 1, scaled by 1.414 to 0 and 0.707. Exponentials: 1 and 2.028, total 3.028. Weights 0.330 and 0.670. Output: 0.33 × [10, 0] + 0.67 × [0, 10] = [3.3, 6.7]. It is the mirror image of the example, as it should be. Also check that your weights add up to 1.
Common mistake
Thinking attention is how the model "focuses like a person". It is a weighted average computed with dot products. Useful, but mechanical.
Read this: "The trophy did not fit in the suitcase because it was too big." What does "it" refer to? You instantly know it is the trophy. Attention is how a model does this: the word "it" scores every earlier word for relevance, finds that "trophy" matches best, and pulls in its meaning.
Check yourself
What produces the attention weights?
Not quite. The weights come from how well the query matches each key.
Why divide by the square root of d?
Not quite. Large unscaled scores push softmax toward one-hot with tiny gradients.
A query matches key 1 much better than key 2. Which description of the output is right?
Not quite. Softmax gives a high but not total weight to the better match, so the output is a weighted blend.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · The full computation, animated.
- Video
Andrej Karpathy · Implements self-attention in code from a toy example.
- ArticleThe Illustrated TransformerArticle
Jay Alammar · Diagram of queries, keys and values.
Spotted a mistake or something unclear? Report it on GitHub.