Harness engineering: real or hype?

What a harness containsCopy Anchor Link

Instructions and rule files such as AGENTS.md. Tools, including MCP servers and the descriptions that tell the model when to use them. Sandboxes where code runs, with limits on what it can reach. Orchestration: subagents, model routing and handoffs. Hooks, which are deterministic code at fixed points such as before a tool call or a commit. Observability: logs, traces and cost metering. Memory and context management, including compaction and resets.

Why it is more than a buzzwordCopy Anchor Link

Anthropic's long-running work found two failure modes that the model alone does not fix. As context fills, models lose coherence and sometimes wrap up early, so the harness adds context resets with a written handoff. And a model grading its own work tends to praise it, so the harness separates the generator from a skeptical evaluator. Both are engineering decisions you can test, and they produced measurable gains.

The honest caveatCopy Anchor Link

Most of this is ordinary systems engineering: retries, permissions, tests, logging and separation of duties. The new part is that the central component is probabilistic, so verification and observability matter more, and changes need evals to prove they help. Google's practice adds meta-debugging: when an agent fails, fix the workflow, not the model. If someone uses the word without naming which of these parts they changed, treat it as marketing.

Try itCopy Anchor Link

Click through the parts in the diagram. Then take your own agent and list which parts exist, which are missing and which failure each missing part allows.

A worked example: one failure, one harness changeCopy Anchor Link

Failure: the agent says "done" but half the feature is missing. Cause: no way to check. Harness change: give it the test suite and require passing tests before it can finish. Failure: it edited a config file it should not touch. Harness change: a permission rule blocking that folder. Each change is small and measurable.

How a hook stops a bad actionCopy Anchor Link

A hook is code that runs at a fixed point, so the model cannot talk its way past it. Trace for a secrets check:

1. The agent finishes a change and asks to run the commit command.
2. The harness pauses the call and runs the hook first.
3. The hook scans the staged files for text shaped like a password or key.
4. It finds one and returns a refusal that names the file.
5. The agent reads the refusal, removes the key, and tries again.

A rule in the prompt, "never commit secrets", works most of the time. The hook runs every time, because the check does not depend on the model remembering. It still only catches what its scan can recognise.

What goes wrong: a harness that grows too heavyCopy Anchor Link

A rules file grows to 300 lines. Some rules contradict each other, and the agent starts ignoring parts of it. The symptom is an agent that obeys rules early in a session and drops them later, as the rules compete with the task for attention and sometimes contradict each other.

To confirm, trim the file to the ten rules that matter. Run the same task ten times with each version and compare the passes. If the short file passes as often or more, the extra rules were noise.

How to tell a checker worksCopy Anchor Link

An evaluator that approves everything adds cost and no safety. Plant a known bug in a copy of the code and ask the evaluator to review it. If it misses an obvious bug, tighten its instructions or give it the tests to run. Repeat with three different bugs before you trust it.

When a harness is the wrong toolCopy Anchor Link

Renaming twenty files once does not need sandboxes, tracing and a second agent. Write a short script and check the result by eye. A harness pays off when the task repeats, runs unattended, or can do real damage.

Common mistakeCopy Anchor Link

Adding harness parts without a way to measure whether they help.