AI Risk Is No Longer Theoretical
- Neil Hare-Brown
- 1 day ago
- 6 min read
Governance Before the Incident, Investigation After It.
Artificial Intelligence is rapidly becoming embedded within everyday organisational activity.
Employees use AI to research, analyse, summarise, draft, classify and recommend. More advanced systems can make decisions, communicate with customers, initiate transactions and increasingly undertake multi-stage activities with limited human involvement.
The risks are already well recognised. They include inaccurate outputs, hallucination, bias, privacy violations, intellectual property infringement, security vulnerabilities, regulatory breaches and potentially harmful actions.
But underlying many of these risks is a more fundamental governance question:
When AI is used to perform work for an organisation, who is responsible for what it does?
This is not simply a question for autonomous AI agents of the future. It exists whenever AI is used in organisational activity today.
AI Is Already Acting with Delegated Authority
It is tempting to distinguish between AI being used merely as a tool and AI being given authority to act independently.
Operationally, that distinction is useful. However, from the perspective of responsibility, it can be misleading.
Consider an employee who asks an AI system to summarise a 100-page technical report and uses that summary when preparing advice for a client. In a recent example it is clear that where a client subsequently alleges that it relied upon erroneous information in such a report and suffered a material loss, simply establishing that "AI was used" would be nowhere near sufficient. An investigation might need to determine which model was used, who prompted it, what sources were supplied, what output it produced, how the output was modified, what verification occurred, who approved publication and whether the erroneous result can be reproduced.
In such cases, the AI has not been authorised to communicate with the client. It has not independently made a corporate decision. Nevertheless, part of the cognitive work underpinning the organisation's eventual advice has been delegated to AI.
The employee is acting on behalf of the organisation, and the AI is being used within that chain of delegated responsibility that flows down:
Organisation → Employee → AI
This is the basis of what I have described as On-Behalf-Of (OBO) Governance.
The governance question is therefore not whether AI is acting on behalf of the organisation. When it is being used to perform organisational activity, it already sits within an OBO chain.
It is important to question what capability, discretion and authority have been delegated to the AI, by whom, and subject to what controls?
There is a continuum. At one end, a human performs a task with relatively minor AI assistance. AI might then progressively summarise, analyse, recommend, decide, act and ultimately undertake complex activities autonomously.
The responsibility flows back up: AI → Employee → Organisation and OBO applies throughout.
What changes is the degree of delegation and consequentiality, and therefore the governance that should accompany it.
Delegation Must Not Make Responsibility Disappear
This distinction becomes increasingly important as organisations adopt more capable AI systems.
An employee may have authority to undertake research but not approve expenditure. Giving an AI system access to organisational systems should not somehow allow that AI to exercise greater authority than the employee through whom its authority derives.
The principle should be straightforward. AI should not acquire authority simply because technology makes an action possible. Its permissible authority should remain bounded by the authority delegated through the organisational chain.
There is an equally important principle:
Capability can be delegated. Authority can be delegated. Accountability cannot simply disappear into the delegation chain.
This becomes particularly important where human oversight exists principally on paper.
Imagine an AI system that produces recommendations which employees accept almost automatically. Formally, a human may remain the decision-maker. Functionally, however, substantial decision-making capability has been delegated to AI.
Good governance needs to recognise the reality of that relationship rather than merely its organisational label.
Governance Standards Provide the Foundations
Organisations are not without guidance.
ISO/IEC 42001 provides an international AI management system standard for organisations developing, providing or using AI systems. ISO/IEC 23894 provides guidance specifically around AI risk management, while ISO/IEC 38507 addresses governance implications associated with organisations' use of AI.
The NIST Artificial Intelligence Risk Management Framework similarly structures AI risk management around four functions: Govern, Map, Measure and Manage.
These frameworks provide extremely useful foundations for AI governance.
But there remains an important issue around the explicit assignment and traceability of responsibility. An organisation can establish policies, risk assessments, controls and oversight mechanisms. But when AI becomes part of the process by which organisational activity is performed, governance must also establish the chain through which capability and authority have been delegated.
That is the contribution OBO Governance seeks to make.
Establishing the Governance Baseline
The answer to AI risk is not preventing organisations from using AI. The potential productivity, innovation and competitive benefits are simply too significant.
The objective should be to make AI governable. A practical starting point is therefore an AI Governance Review.
Before building an elaborate compliance programme, organisations should understand how AI is actually being used.
What AI systems exist? Who is using them? What information can they access? What organisational activities do they support? What decisions can they influence? What actions can they perform? What third parties are involved? What controls exist? Where is human oversight required?
And importantly, for each significant AI-supported activity, where does responsibility sit.
The review can then establish an AI inventory, ownership, policies, risk classifications, approval processes, delegated-authority boundaries, monitoring requirements and controls aligned with frameworks such as ISO/IEC 42001 and the NIST AI RMF.
Governance cannot eliminate AI risk, but it can make the risk manageable.
Insurer Recognition of AI Risk
One indication that AI risk is becoming financially material is the development of insurance products and policy extensions addressing AI-related liabilities.
Insurers increasingly must contemplate losses arising from erroneous AI outputs, intellectual property disputes, privacy breaches, discrimination, professional liability, failure of AI-enabled services and other consequences of organisational reliance upon AI.
Insurance provides an important mechanism for risk transfer, but it introduces another question. When an organisation claims that AI caused a loss, how do we establish what happened?
An insurance policy can transfer some of the financial consequences of an AI incident. However, it cannot establish causation. That requires investigation.
AI Incidents Need Quality Investigations
We have spent decades developing disciplines for investigating conventional technology incidents. Now AI introduces additional complexity.
Suppose an AI-supported process provides damaging advice to a customer.
Was the underlying model and associated vendor responsible? Was the problem introduced by the system prompt? Did retrieval-augmented data influence the answer? Was incorrect organisational information supplied to the model? Was a third-party component involved? Did an employee incorrectly rely upon the output? Were safeguards overridden? Had the model or configuration changed?
And crucially, can the error be reproduced?
Reproducibility may become one of the fundamental principles of AI investigation.
AI systems can be probabilistic. Apparently identical inputs do not necessarily produce identical outputs. Models, prompts, context, retrieved information, configurations and external services may also change.
Investigators therefore need sufficient evidence to reconstruct not simply what an AI system could have done, but what it actually did, under what conditions, using which model and information, and within which chain of delegated responsibility.
That requires evidence preservation, logging, prompt and response histories, model and configuration information, access records, decision trails and potentially forensic examination of connected systems.
Effective AI Investigation: Reconstructing the OBO Chain
This is where governance and investigation become two sides of the same problem; and the solution.
Before an incident, governance should proactively establish:
Who → delegated what → to whom or what → for what purpose → within what limits → subject to what controls.
After an incident, investigation effectively works backwards through that same chain:
What happened → what AI contributed → what information and authority it possessed → what controls operated → who delegated that capability → where responsibility ultimately resides.
Poor governance therefore makes investigation harder and if organisations cannot identify which AI systems were used, preserve relevant prompts and outputs, establish configurations, demonstrate oversight or trace delegated authority, determining causation and responsibility following an incident may become extraordinarily difficult.
Govern. Transfer. Investigate. Learn.
A mature approach to AI risk therefore requires capabilities on both sides of an incident.
Before something goes wrong, organisations need governance: understanding their AI environment, establishing responsibility, controlling delegated capability and authority, and creating the evidence necessary to demonstrate appropriate oversight.
Some residual financial risk can then be transferred through appropriate AI liability insurance. When something does go wrong, organisations need quality AI incident investigation: preserving evidence, reconstructing events, establishing causation, testing reproducibility and determining how the OBO chain operated. The findings from the investigation should then feed back into governance.
This creates a continuous risk-management cycle:
Govern → Transfer → Investigate → Learn → Improve
AI will inevitably make mistakes. Some will be trivial. Others may cause substantial financial loss, regulatory action, litigation or harm to customers and third parties.
The measure of a well-governed organisation will therefore not be whether its AI ever fails. It will be whether the organisation understood how AI was being used on its behalf; understood the capability, discretion and authority being delegated to it; established appropriate controls and accountability; and, when something went wrong, could determine exactly what happened and why.
AI risk does not need to be eliminated to be manageable. But it does need to be governed, and when it fails, it needs to be investigated properly.
