Choosing an embedding model

What differsCopy Anchor Link

Dimensions: the vector length. More dimensions can carry more detail and cost more to store and search. Some models support Matryoshka embeddings, which let you truncate a vector to a shorter length with small quality loss. Context length: the longest text the model accepts per input. Languages: some models are English-only, others multilingual. Domain: code, legal and medical text often benefit from specialized models. Hosting: a paid API, or an open model such as a sentence-transformer you run yourself.

A worked example: a 20-minute bake-offCopy Anchor Link

Collect 30 real questions and note which document answers each. Embed all documents with three candidate models. For each question, check whether the right document is in the top five. Example results (invented for illustration): model X 24 out of 30, model Y 27, model Z 21. Pick Y unless X is much cheaper or faster. Repeat when your documents change character.

What you pay for each choiceCopy Anchor Link

Dimensions cost storage. With invented numbers: 1,000,000 vectors of 1,024 numbers at 4 bytes each is 4,096,000,000 bytes, about 4.1 GB. The same vectors at 256 numbers (for a model that supports shorter outputs) take about 1 GB. Search also gets faster, but each vector holds less detail.

A hosted API means no servers to run, but your text leaves your machine and the provider may retire a model, which forces a re-embed. An open model you host keeps data in-house and the model fixed, but you run the hardware.

Input length matters too. Text longer than the model's limit is cut off by some providers and rejected by others, so check the docs and chunk below the limit.

What goes wrongCopy Anchor Link

The most common failure is a language or domain mismatch. Your documents are in Spanish, the model was trained mostly on English, and results look nearly random. Confirm it by running 10 questions in Spanish and counting how often the right document appears in the top five.

Another quiet failure is a missing instruction. Some models expect a short label telling them whether the text is a question or a document. Leaving it out lowers quality without any error message. The model's documentation says whether it is needed.

A leaderboard score is an average over many tasks. It says little about your contracts, your slang or your product names.

A second example: when a small model is enoughCopy Anchor Link

A team has 200 short FAQ entries and users who copy wording from the page. A small, cheap model gets 28 of 30 test questions right (invented numbers). A large model gets 29. The extra point does not justify a higher bill and slower calls, so they keep the small one. Paying for a bigger model only makes sense when your own test shows it helps.

Try it yourselfCopy Anchor Link

Take 15 questions from real users and the documents that answer them. Test two models, one small and one large, and count how often the right document lands in the top five. Time both. Expect the gap to be small for easy, well-worded questions and larger for vague ones. Write the result down with the date, so you can rerun it when a new model appears.

Common mistakeCopy Anchor Link

Changing the embedding model later without re-embedding everything. Old and new vectors cannot be compared.