Autonomy Is Not the Goal
There is a persistent assumption that the maturity of an agent deployment can be measured by how little human involvement it requires. Remove the human, and you have arrived.
This gets the objective wrong. Nobody buys an agent because it is autonomous. They buy it because a process becomes faster, cheaper, or more consistent — and every one of those benefits evaporates the first time an unsupervised agent does something expensive and wrong.
The useful target is reliability at a known cost of oversight. Autonomy is one input to that, and past a certain point it stops helping. The design question is not "how do we remove the human?" but "where does human judgement earn its keep, and where is it pure friction?"
The Four Ways Agents Actually Fail
Guardrails designed against a vague fear of "the AI going wrong" tend to be badly targeted. It helps to be specific, because the four common failure modes need different countermeasures.
Wrong action. The agent does something it should never have done at all — emails the wrong customer list, deletes a record, orders from the wrong supplier. Cause is usually capability, not reasoning: the action was available when it should not have been.
Right action, wrong scope. The agent does the correct thing far too many times, or at the wrong magnitude. It issues the refund it was supposed to issue, four hundred times. Prompt-level instructions are a weak defence here.
Plausible but wrong output. The most dangerous mode, because nothing looks broken. The summary reads well and misstates a contract term. The reconciliation balances and matches the wrong invoice. Fluency is not accuracy, and reviewers under time pressure conflate the two.
Silent failure. The agent stops doing useful work while appearing to run. Queue depth grows, output quality decays, or a tool has been failing for a week and nobody noticed because the process never surfaced an error.
Note that only the third of these is a "model quality" problem. The other three are systems problems, and they respond to systems fixes.
Constrain Capability Before Behaviour
The most common design error is trying to control an agent with instructions. The system prompt lists what the agent must never do, and everyone feels better.
Instructions are a preference, not a boundary. A boundary is something the agent is structurally unable to cross.
The reliable hierarchy, strongest first:
- Don't expose the tool. An agent that has no delete capability cannot delete. This is the only guarantee that holds unconditionally.
- Constrain the tool's parameters. A refund tool with a hard maximum enforced server-side cannot exceed it regardless of what the agent decides.
- Require approval for the call. The agent may propose the action; a human commits it.
- Instruct the agent. Useful for shaping good behaviour. Never the last line of defence.
The practical rule: for any rule you care about, ask what happens if the agent ignores its instructions entirely. If the answer is unacceptable, the rule belongs at level 1, 2, or 3.
Scoped Mandates
Beyond individual tools, an agent needs a defined envelope for a unit of work. Four dimensions cover most cases.
- Budget — a spend ceiling per task and per period, enforced where the money moves rather than in the agent's reasoning.
- Rate — a cap on actions per interval. This is what turns "issued four hundred refunds" into "issued five refunds and hit a limit".
- Reach — which records, customers, systems, or jurisdictions are in scope at all.
- Time — a deadline after which the agent stops and escalates rather than continuing to try.
Rate limiting deserves particular emphasis because it is cheap, easy to retrofit, and converts a category of catastrophic failures into merely annoying ones. An agent that can do a wrong thing five times is a support ticket. One that can do it unbounded is an incident.
Choosing Approval Gates
Approval gates are the expensive guardrail. Too few and you have unmanaged risk; too many and you have a slow process with a human bottleneck plus the cost of the AI.
Three properties determine whether an action needs a gate:
| Property | Ask | Gate needed when |
|---|---|---|
| Reversibility | Can we undo this in minutes, without anyone outside noticing? | Effectively irreversible |
| Cost of being wrong | Money, legal exposure, reputation, safety | Material |
| Detectability | Would we notice the error on our own, quickly? | Errors are silent or slow to surface |
The instructive combination is the third. An action can be cheap and reversible and still deserve a gate, if a mistake would go unnoticed for weeks. Conversely, an expensive action that fails loudly and immediately may not need one — you will find out and can respond.
Worked examples of the pattern:
- Drafting a customer reply — reversible, cheap, immediately visible. No gate; sample for quality instead.
- Posting a journal entry — reversible with effort, material cost, low detectability. Gate it.
- Sending an external payment — irreversible, material, detectable too late. Gate it, and consider two approvers above a threshold.
- Updating an internal knowledge base article — reversible, low cost, but errors propagate quietly into other answers. Gate it or add a review queue.
Making Human Review Actually Work
A gate that produces rubber-stamping is worse than no gate, because it manufactures a false record of oversight while adding delay.
Review degrades for predictable reasons, and each has a countermeasure.
Volume. A reviewer facing hundreds of approvals per day will approve nearly all of them within seconds. If the gate is not sustainable at your volume, the answer is a narrower gate — not a faster reviewer.
Missing context. Approving "issue refund: CHF 240" is impossible to do well. Approving it alongside the order, the customer's history, the agent's stated reasoning, and the policy clause invoked is a real decision. The reviewer needs what the agent used.
No visible consequence. When reviewers never learn that something they approved was wrong, attention drifts. Sampling approved decisions afterwards and feeding the results back is what keeps review honest.
Approve-only interfaces. If rejecting is harder than approving, approval becomes the default. Rejection must be one click, and it must be cheap to explain why — those explanations are the best training signal you will get.
One structural point: the approver should be someone who would have made the decision anyway. Routing agent output to a reviewer with no domain authority produces a signature, not an assessment.
You Cannot Guardrail What You Cannot See
Every control above assumes you can tell what the agent is doing. Most teams discover their observability gap during an incident.
The minimum worth having before an agent touches production:
- Full traces. For every run: inputs, each tool call with arguments and results, the reasoning, the final output. Attributable to an agent version and an acting identity.
- A regression set. Thirty to a hundred real cases with known-good outcomes, run against every prompt or model change. Without this, you cannot distinguish an improvement from a regression, and you will eventually ship one believing it was the other.
- Outcome metrics, not activity metrics. Tasks completed without escalation, rejection rate at gates, downstream corrections. Volume of agent runs measures nothing about value.
- Alerts on distribution shift. A rejection rate moving from 4% to 15% is the earliest signal that something upstream changed. Silent failure is only silent if nobody is listening.
A Rollout Ladder
Guardrails are not a fixed configuration; they are a position on a ladder you climb as evidence accumulates.
- Shadow. The agent runs and produces output. Nothing is acted on. Compare against what humans actually did.
- Suggest. Output reaches the human doing the work as a proposal. They accept, edit, or discard — and the edit rate is your quality metric.
- Approve. The agent acts, subject to a gate. Track approval rate and time-to-approve.
- Autonomous with sampling. The agent acts unsupervised within a scoped mandate; a percentage of decisions is reviewed after the fact.
- Autonomous with monitoring. Only aggregate metrics and exception handling. Reserved for high-volume, well-understood, bounded actions.
Two rules make the ladder work. Climb only on evidence, using a number agreed in advance rather than a general sense that things are going well. And be willing to descend: a model change, a process change, or a spike in rejections is a reason to step back a rung, not to investigate while running autonomously.
Most valuable deployments settle at step 3 or 4 and stay there — not because they failed to reach full autonomy, but because that is where the economics are best. Recognising that is a sign of a mature programme, not a stalled one.
Deciding where the gates belong is a judgement about your processes, not just your technology. Explore our live showcase of production deployments, or talk to our team about designing oversight that holds up under real volume.