Project: ask questions about your documents

The briefCopy Anchor Link

Pick a small set of documents you know well, such as a product manual or your own notes. The user asks a question and gets an answer with citations to the passages used.

StepsCopy Anchor Link

1. Split the documents into chunks of about 300 tokens with overlap and keep the source and heading as metadata.
2. Embed each chunk and store it in a vector database (pgvector, Chroma or Supabase).
3. Embed a question and fetch the top five chunks.
4. Prompt the model to answer only from the chunks, cite them and say when the answer is not there.
5. Print the retrieved chunks next to every answer so failures are easy to see.

Measure itCopy Anchor Link

Write twenty questions with the document that answers each. Compute recall at five. Try one change at a time: chunk size, adding keyword search, adding a reranker. Keep the changes that improve the number.

Done whenCopy Anchor Link

You have 20 questions, each with the passage that answers it written down. You measured how often the right passage is in the top 5 results and wrote the number down. Every answer shows its source chunk. A question the documents do not cover gets "I could not find that" instead of an invention. You can name two questions it still gets wrong and why.

StretchCopy Anchor Link

Add metadata filters, such as only documents from a chosen year. Add a question you know the documents cannot answer and check that the bot says so.

StarterCopy Anchor Link

Python again. This starter chunks one text file, embeds the chunks locally (no API key needed) and finds the closest ones for a question. Install with pip install sentence-transformers numpy.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def chunks(text, size=200, overlap=40):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

docs = chunks(open("notes.txt").read())
vectors = model.encode(docs, normalize_embeddings=True)  # unit length, so a dot product is cosine similarity

def top(question, k=5):
    q = model.encode([question], normalize_embeddings=True)[0]
    best = np.argsort(vectors @ q)[::-1][:k]
    return [(float(vectors[i] @ q), docs[i]) for i in best]

for score, chunk in top("What is the refund window?"):
    print(f"{score:.2f}  {chunk[:100]}")

Print the scores next to the chunks, as step 5 says. Then add the model call that answers from them, using the pattern in the Calling a model lesson. Swap the in-memory list for pgvector or Chroma once it works.