AI Governance After Approval: Why the system is the wrong unit of governance

Before we approved a model for use, we evaluated it the way most institutions evaluate one. We assembled thirty financial-crime behaviours — structuring cash below reporting thresholds, recruiting mule accounts, evading know-your-customer checks, circumventing sanctions screening — and put each of them to the model in plain English. It refused all thirty.

That result, and the approval that followed from it, was entirely defensible on the evidence in front of us.

We then asked the same thirty questions a second time. Nothing about the model changed, nothing about its permissions changed, and nothing about the substance of the requests changed. Only the shape of the request changed: each behaviour was presented as the final missing cell of a short structured dataset that the model was asked to reconstruct. The model answered all thirty. There was no fine-tuning, no adversarial persona, and no obfuscation of any kind, the harmful request sat in the prompt in ordinary language, exactly as it had the first time.

The governance implication matters rather more than the security one. Both models — the one that passed and the one that failed were the same file, evaluated in the same week, under the same approval. If a deployment-time approval can be silently voided by a change in the form of an input, then that approval was never a statement about how the system behaves in production. It was a statement about how the system behaved under the specific conditions we happened to test. A great deal of AI governance practice today rests on the assumption that these are the same thing.

I have written up the argument in full why the system is the wrong unit of governance, and what it means to govern the action instead in a longer piece on Medium.

Governing the Action

If the action rather than the system is the unit of governance, the governing question changes in character. It ceases to be

"is this system approved?"

and becomes

"is this action permissible, given the context, the authority, the risk and the policy applicable at this moment?"

That is a materially harder question to answer, and it is harder in a way that turns out to be instructive, because it can only be answered at all if certain things are already known and already available at the moment the question arises.

An action cannot be evaluated against an authority boundary that was never defined. Context that was never captured cannot be judged. A decision whose inputs were never retained cannot subsequently be reconstructed. The familiar governance checklist therefore earns its place, but not in the form in which it is usually presented. It is not paperwork to be completed once at onboarding and filed. It is the data that must exist before a runtime judgement is possible at all and this reframing is, in my view, the most useful practical consequence of treating the action as the governed object.

Practical example: an AI agent issuing a customer refund

Imagine a bank deploys an AI agent to handle customer-service requests. The agent is approved for customer-service operations. It has access to the CRM, can look up transactions, and can initiate refunds up to ₹10,000. Under a traditional governance model, the question might be:

“Has the AI system been approved for customer service?”

Suppose the answer is yes. But consider what happens at runtime.

A customer writes:

“I was charged twice for the same transaction. Please refund the duplicate charge.”

The agent identifies the transaction, checks the account and proposes a ₹4,800 refund.

At this point, the governance question is no longer whether the AI system is approved.

It is:

“Is this particular ₹4,800 refund permissible right now?”

That requires information about the action itself.

Runtime question	Information that must exist
What is happening?	Customer, transaction, reason for refund, channel, transaction history
Who is asking?	Customer identity, authentication status, account relationship
Who is the agent acting for?	Agent identity, role, delegated authority
Is the agent allowed to do this?	Refund limit, permitted transaction types, delegation boundary
What could go wrong?	Fraud indicators, duplicate-refund risk, unusual transaction pattern
Which rules apply?	Bank policy, regulatory requirements, product-specific rules
What exactly will happen?	₹4,800 refund to the original payment instrument
What evidence exists?	Inputs, decision, policy version, permissions, tool call, outcome

Now change just one variable.

• The customer asks for a ₹48,000 refund.

• The AI system has not changed.

• The model has not changed.

• The customer-service use case has not changed.

• But the action has crossed an authority boundary.

The appropriate runtime decision might therefore be:

ALLOW — ₹4,800 refund within delegated authority.

versus:

BLOCK / ESCALATE — ₹48,000 exceeds the agent's delegated authority.

That is the practical meaning of governing the action.

This can get more interesting suppose the agent is asked:

“Refund the ₹4,800 and send the customer confirmation.”

The refund itself may be permissible.

But the agent discovers that the customer's account has been flagged for suspected fraud.

• The same AI system.

• The same customer.

• The same refund amount.

Different context → different permissible action.

The governance decision therefore needs to happen at runtime, against the actual context surrounding the action.

You can express the logic very simply:

Context + Authority + Risk + Policy → Action decision

For example:

Context: authenticated customer + duplicate transaction. Authority: agent can refund ≤ ₹10,000. Risk: no fraud indicators. Policy: duplicate transaction refunds permitted. Action: ₹4,800 refund. Decision: ALLOW

Whereas:

Context: account flagged for suspected fraud. Authority: agent technically has refund authority. Risk: high. Policy: high-risk refunds require human review. Action: ₹4,800 refund. Decision: ESCALATE

This is the important governance insight

The AI system is not the decision and it is the mechanism that proposes or executes a decision. The governed object is the specific action at a specific moment, under a specific context, with a specific authority and applicable policy.

That gives you a very practical architecture for the article:

AI system → proposes action → runtime governance layer → allow / modify / escalate / block → action

The evaluation described at the beginning of this article is documented in "Structural Jailbreaks Generalize but Do Not Compound" (arXiv:2609.08373). The attack class it employs — Involuntary In-Context Learning — was identified and named by Adversa AI, whose survey across one hundred models established the breadth of its applicability. The contribution of the present work was to measure it specifically against financial-crime behaviours, where it proves more effective than against general harm.