Error analysis

The methodCopy Anchor Link

Collect a sample of real traces, the full record of each run including inputs, tool calls and outputs. Read them one by one and write a short note on what went wrong in each. After around fifty, group the notes into categories such as "wrong tool", "ignored the instruction", "stale source". Count each category. The largest ones are what you fix first, and each category becomes an automated check.

Why it worksCopy Anchor Link

Generic metrics such as "helpfulness score" miss your product's specific failures. Reading traces finds them. It also builds the intuition to write good graders and good prompts. Do it again after each round of fixes, because new failure modes appear once the old ones are gone.

A worked example: fifty tracesCopy Anchor Link

Open fifty failed conversations. Next to each, write one line: "ignored the date filter", "quoted an old policy", "called the wrong tool", "answered a question it should refuse". Group the lines. Counts: old policy 17, wrong tool 11, ignored filter 9, other 13. Fix "old policy" first (probably stale documents), then rerun and read another fifty.

Make each category a testCopy Anchor Link

For every category, add two or three real examples to the test set. The same mistake can then never return unnoticed.

How to write a good noteCopy Anchor Link

One trace often has several problems. Note the first thing that went wrong, because later errors often follow from it. If the search returned an old policy, the answer was wrong and the tone was confident. Label it "old policy", not three separate failures.

Write what happened, not a judgment. "Quoted the 2022 return window" is useful. "Bad answer" is not, and it cannot be fixed or counted.

When to stop readingCopy Anchor Link

Stop when new traces stop producing new categories. Invented numbers: the first 30 traces give six categories, the next 20 add one. That is a good sign you have seen most of the common failures.

What goes wrong: categories nobody agrees onCopy Anchor Link

Symptom: the biggest group is "other" or "bad answer". Confirm by having a colleague label the same 20 traces without seeing your labels. If you differ on many of them, the category names are too vague. Split them until two people pick the same label.

Another trap is reading only conversations with a thumbs-down. Unhappy users are a biased sample, so also read some random ones.

When reading traces is not enoughCopy Anchor Link

If failures are rare, say 1 in 500, a random sample of 50 may show none. Pull traces that a cheap code check or a user report already flagged. If the product has no users yet, there are no traces. Write realistic inputs yourself first.

Common mistakeCopy Anchor Link

Starting with abstract scores like "helpfulness 7 out of 10" before reading a single trace.