Chunking documents

Why chunkCopy Anchor Link

An embedding of a whole book blurs everything into one point. A chunk that covers one idea matches a question about that idea. Chunks must also fit in the prompt next to others, so they cannot be huge. Too small, and a chunk loses the context needed to make sense of it, such as which product or date it refers to.

Common strategiesCopy Anchor Link

Fixed size: every 500 tokens with a 50-token overlap so ideas cut at a boundary appear in two chunks. Structural: split on headings, paragraphs or code functions so chunks follow the author's own units. Semantic: split where the topic changes, detected by embedding sentences and cutting at large jumps. Hierarchical: index small chunks for matching but return the larger parent section for reading. Contextual: prepend a one-sentence note to each chunk saying where it came from.

Practical tipsCopy Anchor Link

Start with structural splitting at 200 to 500 tokens and tune on your evals. Keep metadata with every chunk: source, title, section, date and permissions. Inspect actual chunks, since bad splits are easy to see and common.

A worked example: overlapCopy Anchor Link

Text: "... demand rose. Prices increased by 4 percent because supply was limited. The central bank ..." Chunk 1 ends after "Prices increased by" and chunk 2 starts at "4 percent because". Neither holds the whole fact. With a 50-token overlap, chunk 2 starts a few sentences earlier and contains it whole.

How a fixed-size splitter worksCopy Anchor Link

Take a document of 1,000 tokens, a chunk size of 400 and an overlap of 50. Each chunk starts 350 tokens after the previous one (400 minus 50). The chunks are:

1. Tokens 1 to 400.
2. Tokens 351 to 750.
3. Tokens 701 to 1,000, which is shorter because the document ends.

The overlap means the index stores about 14 percent more text than the document holds (400 divided by 350). That is the price of keeping boundary sentences whole.

Trade-offs in sizeCopy Anchor Link

Bigger chunks keep more context but match less precisely, and each one uses more of the prompt. Five chunks of 500 tokens put 2,500 tokens in the prompt. Five chunks of 200 tokens put 1,000. Smaller chunks are cheaper to send and sharper to match, but they lose the surrounding facts. Hierarchical chunking tries to get both.

A second example: when not to chunk muchCopy Anchor Link

A help center has 300 FAQ entries, each 80 tokens, one question and one answer. Each entry is already one idea, so splitting further would only cut answers in half. A 2-page memo fits in a prompt whole, so it needs no retrieval at all. Chunking earns its place on long documents with many topics.

What goes wrongCopy Anchor Link

Symptoms you can see by reading real chunks:

1. A chunk that starts mid-sentence, so the split ignored the text's structure.
2. A chunk that says "it" or "the product" with no hint of which one.
3. A table row separated from its column headers.
4. A chunk that mixes two topics, so it matches both questions badly.

To confirm, print ten random chunks and read them as if you were the model. If a chunk makes no sense alone, it will not retrieve well either.

Try it yourselfCopy Anchor Link

Take one long document and 15 questions with known answers. Split it at 200 tokens and again at 600. For each question, check whether the right text appears in the top five chunks. Expect the small size to win on precise facts and the large size to win on questions needing context. Pick by your numbers.

Common mistakeCopy Anchor Link

Choosing the size by guesswork and never measuring. Try two sizes and compare recall on your test questions.