Guardrails Before Autonomy: A Practitioner's Governance Model for Agentic AI

There is a moment in every agentic AI conversation where the language shifts, and I have learned to listen for it.

For the first twenty minutes everyone talks about what the agent will suggest. Recommendations. Drafts. Proposals for a human to review. Everybody is comfortable. Then somebody points out — correctly — that if a person still has to check every output, most of the benefit has evaporated, because checking is nearly as slow as doing.

So the language changes. The agent will handle the routine cases. It will process the straightforward ones. It will close the loop where the confidence is high.

Those are all reasonable things to want. They are also the point at which the system stops being a copilot and becomes something that takes actions in your business with real consequences, and I have watched a lot of programmes cross that line without anybody saying out loud that they had crossed it.

The distinction I hold to is simple: a system that suggests needs to be useful. A system that acts needs to be governed. And the governance has to exist before the scope is granted, not after the first incident, because after the first incident you are not designing controls, you are negotiating with a risk committee that no longer trusts you.

Why the usual answer is not an answer

Ask most enterprise programmes what their AI governance is and you will get some combination of: a policy document, an approved-use-case list, a mandatory training module, and the phrase “human-in-the-loop.”

None of that is worthless. None of it constrains an agent’s behaviour either.

The problem with “human-in-the-loop” as it is usually deployed is that it names a principle without specifying a mechanism. In practice it very often means a notification goes somewhere. Somebody, in theory, reviews it. In a busy quarter, nobody does, and the queue becomes a formality that everybody clears in bulk on a Friday afternoon. I have seen this in production more than once. The oversight existed on the architecture diagram and had been socially discontinued.

Real governance answers which named role, at what threshold, in which system, within what time, and with what happens on no response. If you cannot fill in those five blanks, you do not have oversight. You have an agent running unsupervised, with a comforting paragraph about oversight in the documentation.

The five questions

This is what I take into the design session. It is deliberately short — a long framework does not get used — and every item is here because I have watched something go wrong for the want of it.

1. What can it read?

Scope of retrieval, stated positively as an allow-list rather than negatively as exclusions. Exclusion lists rot: somebody adds a new source, nobody updates the list, and the agent quietly gains access to compensation data on a Tuesday. Allow-lists fail closed, which is the direction you want them failing.

This is also where data residency and segregation-of-duty questions get settled, and it is far cheaper to settle them here than in a security review three weeks before go-live.

2. What can it do, and what is the worst realistic outcome if it is confidently wrong?

Not the worst conceivable outcome — you can catastrophise any system into paralysis. The worst realistic one. There is an enormous difference between:

• an agent that drafts an exception note into somebody’s queue,
• an agent that reclassifies a line item in a subledger,
• and an agent that releases a payment to a vendor.

The first is a typo risk. The third is a controls issue that involves your auditors. Same technology, three completely different governance conversations, and the mistake I see is treating them as one because they are all “the AI project.”

Write the worst realistic outcome down in one sentence. That sentence determines everything downstream, and it is remarkable how often nobody has said it plainly before.

3. Where is the threshold, and who owns the decision above it?

Give the agent a band it operates in freely and a band where it must stop and ask. A variance under a stated amount, in a stated category, with a stated confidence — handled. Anything outside that band goes to a specific role.

Two things matter here. The threshold must be a business parameter that a business owner can change without a development cycle, or it will be set once and never revisited. And the escalation must go to a role with capacity, not to the most senior person available, or it silently becomes the bulk-approval queue I described earlier.

4. What does the audit trail capture, and who reads it when nothing is wrong?

Everybody builds logging. Almost nobody builds review. An audit trail that is only opened after an incident is forensics, not governance.

The version that works is a small, boring, recurring review: a sample of the agent’s decisions, looked at by the process owner, on a cadence, with the explicit question “would you have done this?” That is where you catch the class of failure that never triggers an alert — the agent being consistently, reasonably, quietly wrong in a way that matches the data it was given.

5. How do you turn it off, and who is allowed to?

The most frequently skipped question, and the one that most determines whether risk officers will sign at all. A named role, a mechanism that does not require an emergency change request, a defined state the process falls back to, and a rehearsal that proves the fallback still works.

If the manual process has atrophied to the point where switching off the agent means the work simply stops, then you have not automated a process — you have replaced it, and the risk profile is completely different from the one in your business case.

Where I have overcorrected

I want to be straight about the failure mode on my side of this table, because it is real and I have caused it.

Governance can become the thing you do instead of shipping. Five questions is a design session. Five questions plus a steering committee plus a review board plus a quarterly attestation cycle is a programme that will produce documents for eighteen months and nothing a customer can use. I have been part of one of those. The controls were excellent. There was nothing to control.

Worse, over-tight governance does not reduce AI usage — it relocates it. When the sanctioned path takes three months of approvals for a use case a business team could stand up in an afternoon, they stand it up in an afternoon, in a browser tab, on data they exported themselves, with no logging, no allow-list, no threshold and no off switch. Every control I described above is absent in that version. The strict regime produced precisely the ungoverned outcome it existed to prevent.

So the calibration I now aim for: the weight of the governance should match the blast radius, and nothing else. An agent drafting text into a queue should clear in a single session with a one-page answer to the five questions. An agent touching the ledger should get the full treatment, including the rehearsal. Applying the ledger-grade process to the drafting case is not caution. It is a tax that pushes people toward shadow tooling.

The short version

I have started closing these sessions with the same request, and it is a decent test of whether we are ready.

Tell me, in one sentence each: what it reads, what it does, what happens if it is wrong, who catches that, and who can stop it.

If a customer can answer all five without hedging, the use case is usually production-ready and the risk conversation is short. If they cannot answer two of them, we are not blocked — we just found this session’s work. Either way it is a better outcome than the alternative, which is discovering the answers from an incident report.