What are embeddings?

The ideaCopy Anchor Link

Computers cannot compare meaning directly, but they can compare numbers. An embedding model reads a sentence and outputs a vector, which is a list of numbers, often 384 to 3,072 of them. The model is trained so that sentences with similar meaning get vectors that land close together. "How do I reset my password?" and "I forgot my login" end up near each other. "Best pizza in Rome" ends up far away.

A tiny exampleCopy Anchor Link

Imagine a 2-number embedding where the first number means "about animals" and the second "about vehicles". "cat" might be [0.9, 0.0], "dog" [0.8, 0.1], "truck" [0.0, 0.9]. The cat and dog vectors are close. Cat and truck are far apart. Real embeddings have hundreds of dimensions that are not human-readable like this, but the geometry works the same way.

Plot the three as points on a graph, with "animals" along one axis and "vehicles" along the other. Cat and dog land in the same corner. An arrow from the middle of the graph to each point is the vector's direction, so "pointing the same way" just means "landing in the same corner".

Where embeddings are usedCopy Anchor Link

Semantic search (find documents by meaning), retrieval for RAG, clustering similar items, recommendations, duplicate detection and classification with few labels.

Try itCopy Anchor Link

Use the demo above to drag two arrows and see how the angle between them sets the similarity score.

A worked example: three sentencesCopy Anchor Link

A: "How do I reset my password?" B: "I forgot my login details." C: "What is the weather in Rome?" Scored with cosine similarity (a closeness score where 1 means the same direction and 0 means unrelated), A to B might get 0.82 and A to C 0.08. No shared keyword between A and B, yet the model knows they are about the same thing. That is why a help center search can answer "can't sign in" with the password-reset article.

How a model learns where to put textCopy Anchor Link

Embedding models are usually trained on pairs of texts that belong together, such as a question and the passage that answers it. Each round nudges the model's weights so the pair moves closer. The same round pushes the pair away from unrelated texts in the same batch (a batch is a group of examples handled together). After millions of rounds, topics form clusters.

Here is one round with invented numbers:

1. The pair is "reset password" and "forgot login". The model scores them 0.20.
2. It scores "reset password" against "pizza in Rome" at 0.35, which is wrong.
3. Training shifts the weights a little so the first score rises and the second falls, say to 0.25 and 0.30. The order is still wrong, but the gap has narrowed.
4. Millions more rounds repeat this until the related pair clearly wins.

A second example: where embeddings blurCopy Anchor Link

Take "The hotel was clean" and "The hotel was not clean". The words are almost identical and the meanings are opposite. Embeddings often rate such a pair as very close, because both are about a hotel and its cleanliness. The same happens with "refund within 30 days" and "refund within 3 days": the number changes and the vector barely moves.

Embeddings capture topic well and fine detail poorly. Negations, numbers, rare names and error codes are the weak spots. Later lessons cover the fixes: keyword search catches exact strings, and a reranker reads question and passage together.

How to tell it is workingCopy Anchor Link

Write 10 pairs of sentences: five that should match and five that should not. Embed all of them and compute the cosine similarity for each pair. Every matching pair should score above every non-matching pair. If one does not, note what kind of pair failed (negation, number, name) so you know the model's blind spot.

Compare scores only within one model. A score of 0.82 from one model has no fixed meaning in another, because each model spreads its scores differently.

Common mistakeCopy Anchor Link

Expecting an embedding to answer a question. It only finds related text, and a model still has to read that text and respond.

The math, when you are readyCopy Anchor Link

Closeness between two embeddings is measured with a dot product or cosine similarity. Vectors and similarity works through both with small numbers. The next lessons here explain what you need, so you can skip it for now.