Tracing and observability

What to captureCopy Anchor Link

For each run: the inputs, every model call with its prompt and output, every tool call with arguments and result, timing, token counts and cost. Tag runs with the version of the prompt and model. Trace tools show them as a tree you can open and read, and let you search for failures, slow runs and expensive runs.

What you do with itCopy Anchor Link

Sample traces for error analysis, set alerts on error rate and cost, replay a failing run with a changed prompt, and turn real failures into eval cases. Be careful with privacy: traces contain user data, so redact and set retention limits. Hosted options include LangSmith and Helicone. PostHog and Arize add product analytics and monitoring. Track cost and latency per request, and alert on production drift.

A worked example: finding a cost spikeCopy Anchor Link

Cost per day jumps from $40 to $210. In the trace tool, filter by day and sort by tokens. One conversation made 63 tool calls in a loop reading the same file. The cause: a tool that returned an error the model kept retrying. Fix the error message, add a retry cap. Without traces this would take days to find.

What a trace looks likeCopy Anchor Link

A trace is a tree of spans. A span is one timed step. Invented example for a refund question that took 6.2 seconds in total:

1. Retrieval took 0.4 seconds.
2. The first model call took 2.1 seconds and chose a tool.
3. The tool call took 0.3 seconds.
4. The second model call took 3.2 seconds and wrote the answer.

The four spans add up to 6.0 seconds, and 0.2 is your own code. The slow part is the final model call, so shortening its input is the first thing to try.

Why long runs cost so muchCopy Anchor Link

Each model call re-sends the whole history. If every step adds 500 tokens, call number n sends about 500 times n tokens. Ten calls send 500 times 55, which is 27,500 tokens. Twenty calls send 500 times 210, which is 105,000. Twice the steps cost nearly four times the input, unless caching helps.

This is why one looping run can dominate a day's bill. Sorting traces by tokens finds it at once.

What goes wrong: gaps in the traceCopy Anchor Link

Symptom: the trace shows a wrong answer but no cause. Often tool errors were caught and not recorded. To confirm, take one failing run and try to explain it using only the trace. If you must guess, a span is missing.

Common mistakeCopy Anchor Link

Logging only the final answer. When something goes wrong you need the steps.