The AI Evidence Gap

Every day, AI systems make decisions that change people's lives:

• A bank's AI declines a mortgage application
• An insurer's AI denies a claim
• A hospital's AI recommends a treatment path
• A government agency's AI flags an applicant for review

When a human makes these decisions, there is a paper trail — notes, files, a record that can be reviewed and appealed. When an AI makes these decisions, there is often nothing: no verifiable record of what was asked, what was returned, or whether either has been altered since.

If the decision is challenged, what evidence exists?

Recent events across the AI industry have highlighted a recurring challenge. When questions arise about model behaviour, safety, compliance or accountability, customers, regulators and the public are often dependent on evidence controlled by the model provider itself.

The issue is not whether providers act in good faith. The issue is whether independent verification is possible.

In mature industries, critical decisions are supported by tamper-evident records, audit trails and independently verifiable evidence. Yet AI systems are increasingly involved in decisions with legal, financial and societal consequences while the underlying evidence often remains inaccessible, unverifiable or controlled by a single party. This creates what I call the AI Evidence Gap.

Why Existing Governance Is Not Enough

Most AI governance initiatives focus on controls — policies, risk assessments, model reviews, red-teaming, human oversight. These controls are important, but they answer a different question.

Controls attempt to prevent failures. Evidence helps explain what happened when prevention fails.

Every mature industry eventually discovers this distinction. Aviation has safety procedures and flight recorders. Finance has controls and transaction ledgers. Cybersecurity has prevention systems and forensic logs. AI is rapidly developing the control layer, but the AI Evidence Layer remains largely missing.

Imagine a future dispute involving an AI system. A customer claims the model provided harmful advice. The provider disputes the claim. A regulator requests evidence. An insurer asks whether safety controls were operating at the time. A court seeks to establish the facts. What evidence exists?

Today, the answer is often logs, screenshots and internal investigations — all potentially useful, but ultimately dependent on trust in the organisation providing them. And no provider can close this gap on its own: a party verifying its own behaviour is self-referential by design, not through bad faith but by structure.

As AI systems become more capable and more deeply embedded into society, governance alone is insufficient. Controls matter, but so does evidence that those controls were applied.

The future of AI accountability will not be built solely on policies, audits and assurances. It will increasingly depend on independently verifiable evidence. The organisations best positioned for that future will not simply be those with the strongest governance frameworks — they will be the ones capable of proving what happened.

The challenge is not determining who should be trusted. The challenge is whether trust should be required at all.