Vectors and similarity
A vector is a list of numbers. The dot product measures how much two vectors point the same way. A large share of modern AI is this operation repeated.
Dot product and cosine similarity
a · b = 3.86 | cosine = 0.64
Partly aligned. Lengths: a = 3, b = 2.
Words on this page (9)
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- dot product
- : Multiply matching entries of two vectors and add them up. Large when they point the same way.
- prompt
- : The input text you give a model, including instructions, examples and data.
- embedding
- : A list of numbers that represents the meaning of a piece of text or an image, so similar things sit close together.
- retrieval
- : Finding the most relevant pieces of text for a query.
- agent
- : A model run in a loop that chooses its own tools and steps until the task is done.
- cosine similarity
- : A score from -1 to 1 for how much two vectors point the same way, ignoring their length.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- neuron
- : One unit in a network: it multiplies inputs by weights, adds a bias and applies an activation function.
Before you start
This chapter is the math underneath everything you have used so far: , , and . You have already seen what these things do. Here you see why they work, using small numbers you can do by hand. Take it one lesson at a time. If a formula gets heavy, read the worked example and the picture first, and come back to the formula later.
The idea
Represent a thing as numbers. A word, an image or a sentence becomes a such as [0.2, 0.9, -0.1]. The multiplies matching entries and adds them: a · b = a₁b₁ + a₂b₂ + … Think of each vector as an arrow from the middle of a graph to the point it names, so its direction is where the arrow points. Large and positive means similar direction, near zero means unrelated, negative means opposite.
divides out the lengths so only direction counts: cos(a, b) = (a · b) / (|a| |b|), where |a| is the square root of the sum of squares. It always falls between -1 and 1.
Example
Take a = [1, 2] and b = [2, 1]. The dot product is 1×2 + 2×1 = 4. The lengths are both √5 ≈ 2.236, and √5 × √5 = 5. So cosine is 4 / 5 = 0.8, quite similar. For a = [1, 0] and c = [0, 1], the dot product is 0, so cosine is 0: unrelated.
Where you will see it
Embedding search ranks documents by cosine similarity to the query. Later you will meet the same operation inside and inside every , but you do not need those yet. Learn this one operation well.
Try it
Drag the two arrows in the demo above and watch the dot product and cosine change.
Worked example: recommending a song
You: [fast 0.9, acoustic 0.1, sad 0.2]. Song A: [0.8, 0.0, 0.1]. Song B: [0.1, 0.9, 0.9]. Dot with A: 0.9×0.8 + 0.1×0 + 0.2×0.1 = 0.74. Dot with B: 0.9×0.1 + 0.1×0.9 + 0.2×0.9 = 0.36. Song A fits you better. The songs are all about the same size, so the plain dot product is fair here. Search engines, recommendations and chatbots all use this idea at much bigger scale, with hundreds of numbers instead of three.
How length and direction separate
Every vector has a length, written |a|. For [3, 4] it is √(3² + 4²) = √25 = 5. Dividing each entry by the length gives [0.6, 0.8], a vector of length 1 that points the same way. Cosine similarity is the dot product of two such length-1 vectors, which is why it ignores size. Doubling a vector, from [1, 2] to [2, 4], doubles its dot products but leaves its cosine with [1, 2] at exactly 1. Flip it to [-1, -2] and the cosine is -1. A vector like [2, -1] gives dot product 0 with [1, 2], so the cosine is 0.
A second example: when length misleads
Your search query is a = [1, 1]. Document X = [1, 0.9] is short and almost the same direction. Document Y = [10, 2] is long and leans mostly one way. Raw dot products: a · X = 1.9 and a · Y = 12, so Y wins. Lengths: |a| = 1.414, |X| = 1.345, |Y| = 10.198. Cosines: 1.9 / (1.414 × 1.345) = 0.999 for X and 12 / (1.414 × 10.198) = 0.832 for Y. Cosine picks X, which really matches. If a system already scales every vector to length 1, the dot product and cosine are the same number, and the cheaper dot product is enough.
What goes wrong
Embedding vectors (lists of numbers that stand for meaning) are learned from how words are used, not from a dictionary. Words like "hot" and "cold" appear in similar sentences, so they can score a high cosine even though they are opposites. A high score means "used in similar places", not "means the same". You can confirm this by printing the nearest neighbors of a few words and reading them. A second failure is an all-zero vector, whose length is 0, so the cosine divides by zero and returns an error or not-a-number.
Check it by hand
Pick a = [3, 4], b = [6, 8] and c = [4, -3]. Work out a · b, a · c and the three lengths. You should get a · b = 50, a · c = 0, lengths 5, 10 and 5. Then the cosine of a and b is 50 / (5 × 10) = 1, and the cosine of a and c is 0. Same direction and a right angle, found with arithmetic alone.
Common mistake
Reading a dot product as a percentage. It has no fixed maximum and grows with vector size, so only cosine similarity stays between -1 and 1.
A music app wants to suggest songs. It describes you as a list of numbers: how much you like fast songs, how much you like acoustic songs, how much you like sad songs. It describes each song the same way. To decide if a song fits you, it multiplies the matching numbers and adds them up. Big total, good match. That multiply-and-add is the dot product.
Check yourself
a = [1, 2] and b = [3, 0]. What is the dot product a · b?
Not quite. Multiply matching entries and add: 1 × 3 + 2 × 0 = 3.
Two embeddings have a cosine similarity near 0. What does that mean?
Not quite. Cosine 1 means the same direction, 0 means unrelated and -1 means opposite.
You compare a short note and a very long article by raw dot product, and the long one always wins. What is going on?
Not quite. Length inflates the dot product. Cosine similarity divides the lengths out so only direction counts.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · Essence of linear algebra, chapter 1.
- Video
3Blue1Brown · Why multiplying and summing measures alignment.
- ArticleThe Illustrated Word2vecArticle
Jay Alammar · How words become vectors you can do math on.
Spotted a mistake or something unclear? Report it on GitHub.