Guardrails and human approval
Decide which actions an agent may take alone, which need approval and which are never allowed.
Words on this page (7)
- agent
- : A model run in a loop that chooses its own tools and steps until the task is done.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- hook
- : A script the harness runs automatically at a fixed point, such as before a tool call. It enforces rules in code.
- prompt
- : The input text you give a model, including instructions, examples and data.
- allowlist
- : A list of the only actions or sites something is permitted to use. Everything not on it is refused.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- git
- : The standard tool for tracking changes to code.
The approach
Sort actions by risk and reversibility. Reading files is low risk. Editing files in a branch is reversible. Sending an email, spending money, deleting data or deploying are costly to undo. Allow the first group, the second and require approval for the third. Enforce with code: permission systems, allowlists of commands, sandboxes and . A line in a is a request, a hook is a rule.
Make approval useful
Show what will happen in plain terms and what the tried before. Avoid approval fatigue: if every action asks, people click yes without reading. Group low-risk actions and surface only real decisions. Keep an audit log of what ran and who approved it.
A worked example: sorting actions by risk
Read a file: allow. Edit a file in a branch: allow and log. Send a customer email: ask first. Delete data or spend over a limit: require approval and a reason. The sort uses two questions: how bad is a mistake, and can it be undone?
Approval fatigue in numbers
Invented numbers. An agent asks for approval 50 times a day. 49 are harmless file edits and one is an email to an outside address. By the 50th, the person has pressed yes 49 times without reading and approves the bad one too.
Put 40 of those low-risk actions on an and log them. Now the person sees 10 requests, and the risky one stands out.
What goes wrong: the approval shows the wrong thing
The dialog says "Sending the weekly report", written by the model. The real call sends the file to a personal address. The symptom is approved actions that surprise the approver. Confirm it by comparing the text on screen with the arguments the actually received. Build the approval screen from those arguments in code: recipient, subject, attachment. Do not use the model's own description of them.
Allowlists need care
Suppose you allow any command that starts with . Then "git push --force" passes, and so does "git status; rm -rf notes". Match the full command against a list of exact patterns, or break the line into parts first and check each part.
Try it yourself
Write "never run rm" in the prompt, set no hook, and ask the agent to clean up a test folder in a way that needs rm. Then add a hook that rejects any command containing rm and repeat. Expect the prompt-only version to be ignorable under pressure and the hook version to always block and write a log line.
Common mistake
Putting the rule in the prompt. Enforce it with permissions and hooks.
A company lets an agent issue refunds. For amounts under $10 it acts alone. Between $10 and $100 it asks a person to approve with one click. Above $100 it only drafts. After a month the team sees that the person approved 98 percent of the middle tier. They first check that reviewers read each request and did not just click through, then raise the automatic limit to $25. Trust was earned with data.
Check yourself
Which action should usually need human approval?
Not quite. It is outward-facing and hard to reverse.
Why is a hook stronger than a prompt rule?
Not quite. Prompts request, hooks enforce.
An agent asks for approval on every single action and people click yes without reading. What is the fix?
Not quite. Approval fatigue makes checks useless. Reserve them for meaningful decisions.
You got 0 of 3 right.
Go deeper
- Docs
Anthropic · Allow, ask and deny rules for tools.
- Docs
Anthropic · Deterministic enforcement at lifecycle points.
- Repo
NVIDIA · Programmable guardrails for LLM apps.
Spotted a mistake or something unclear? Report it on GitHub.