Rewriting the question before searching

What it isCopy Anchor Link

People do not write search queries. They write "and for gift cards?" in the middle of a chat, "it's not working" with no detail, or two questions in one. A rewrite step uses a small model call to turn the raw text into something retrieval can use. Common techniques:

Standalone rewrite: use the chat history to remove pronouns and fill in the gaps. Multi-query: write three different phrasings, search each, and merge the results. Decomposition: split a two-part question into two searches. Step-back: ask the broader question behind the specific one. HyDE: have the model write a short hypothetical answer, then search with that text, because an answer looks more like a document than a question does.

A worked exampleCopy Anchor Link

Chat: "What is the refund window?" The bot says 30 days. The user asks: "And for gift cards?" Searching that text returns random gift card pages. The rewrite step, given the history, produces: "What is the refund policy for gift cards?" and now the right chunk is found. For "How do I recover my account?", a multi-query rewrite searches "reset password", "locked out of account" and "forgot username", then merges the three lists with reciprocal rank fusion.

How to write the rewrite stepCopy Anchor Link

The rewriter is a small prompt with clear rules:

1. Give it the last few chat turns and the new message.
2. Ask for one standalone question with no pronouns.
3. Tell it to keep every name, number, date and place exactly.
4. Tell it to return the question unchanged if it is already clear.

Keep the original question too. If the rewrite goes wrong, you can fall back to searching the user's own words.

A second example: when a rewrite hurtsCopy Anchor Link

A user types "ERR_4021 refund". A helpful rewriter might turn it into "problems with payment errors". The exact code is gone, so keyword search can no longer find the one page that mentions it. Short, precise queries with identifiers are best left alone. HyDE has a related risk: the invented answer may contain wrong facts, and those wrong facts then steer the search toward the wrong documents.

What goes wrongCopy Anchor Link

The symptom is recall that drops for questions that used to work. Another sign is a rewrite that answers instead of asking, such as returning "30 days" in place of a question. A third is cost: multi-query runs three searches instead of one, so latency and spend rise.

How to tell it is workingCopy Anchor Link

Log the original question and the rewrite side by side for 20 real conversations and read them. Then compute recall at 5 on your test set with and without the rewrite step, split by question type: follow-ups, vague questions and clear questions. Keep the step where it helps and skip it where it does not.

When to use itCopy Anchor Link

Add it when your test set shows misses on follow-ups, vague questions or vocabulary mismatches between users and documents. It costs one extra model call, so use a small fast model and skip it for clear, self-contained questions. Measure recall before and after.

Common mistakeCopy Anchor Link

A rewrite that drops a constraint. "Refund policy for orders over $100 in Canada" must not become "refund policy". Tell the rewriter to keep every name, date, number and place.