IAIntelligence Architecture
Artificial Intelligence

What Makes an AI Output Worth Trusting

In the Horizon case, a disputed accounting shortfall became a debt and eventually a suspicion. What mattered was whether it could still be examined and corrected effectively.

Oliver Wrede
Oliver Wrede·4 min read
A close view of an impasto oil painting. A dark machine displays a polished blank tablet. Behind it, two rows of small green marks wind across the surface; a small indigo block bends over the trail.
A close view of an impasto oil painting. A dark machine displays a polished blank tablet. Behind it, two rows of small green marks wind across the surface; a small indigo block bends over the trail.

The Horizon accounting system displayed a shortfall in British post office branches when declared holdings differed from the totals derived from recorded transactions. The branch operator could not mark the amount as disputed in the system: to open the next trading period, the operator had to cover it or settle it centrally as a sum owed to the Post Office. The High Court later found that challenged shortfalls were treated as debts.1

An accounting mismatch could thus acquire the status of a debt and eventually support an allegation of theft, fraud, or false accounting. In 2021, the Court of Appeal quashed 39 convictions in which assumed reliability of Horizon data had been essential; it dismissed three appeals with materially different evidence.2

The significance for organizational AI does not depend on claiming that Horizon resembled a language model. It lies in the process: a technical output can acquire more authority than its evidence supports when affected people cannot inspect its basis and their challenge cannot reach the decision in time.

What the case contributes to the argument

A correctly calculated difference establishes only that declared holdings do not match the balance derived from the system’s records. It establishes neither that those records reflect the branch accurately, nor that money is missing or a particular person is responsible. For the older Horizon versions it examined, the High Court also found a material risk that errors in data entry, transfer, or processing could affect branch accounts.1 That is not a claim about every entry or every later state of the system.

The decisive issue was unequal inspectability. Fujitsu had privileged means of intervening in transaction data, while the Post Office had reports unavailable to branches; the court found some controls and records inadequate. Audit data could resolve disputed corrections but depended on access from the organizations whose system and claim were under review. The Court of Appeal accordingly distinguished whether Horizon had been essential to the evidence in each case.

Trust therefore requires more than technical consistency or the possibility of a later investigation. Counterevidence must reach the responsible person while the outcome can still change. A perfect audit trail offers little protection once a debt, sanction, or decision is practically irreversible.

What transfers to AI

AI-generated outputs introduce additional transitions between evidence and claim. A cited source may exist without supporting the sentence attached to it. A summary may contain accurate statements while omitting an objection that changes the decision. Even a faithful summary does not automatically warrant the recommendation that follows. This article makes that transfer; the British courts did not address AI.

The mechanism implies four requirements for organizational use:

  1. Access to evidence: Material claims lead to retrievable, versioned sources. Reviewers can inspect evidence beyond the interface and summary under review.
  2. Controlled intervention: Privileged changes to the system or its data are limited, attributable, and recorded. Actual behavior is monitored after deployment.
  3. Effective challenge: An objection reaches a responsible person before the decision is irreversible. That person can override the output, pause its use, and initiate an investigation.
  4. Proportionate assurance: Review effort reflects the particular use and the consequences of error. Trust attaches to a use under known conditions, not to a system in the abstract.

The voluntary NIST AI Risk Management Framework assigns related responsibilities across the system lifecycle. It calls, where appropriate, for independent assessors, feedback and appeal processes, and clear responsibility for overriding or correcting outputs, deactivating systems, and responding to incidents.3 These measures do not certify any individual output.

Even a careful process can fail. Evidence remains incomplete, independent reviewers may share the same mistaken premise, and an appeal process may exist on paper without practical force. Warranted trust is therefore not demonstrated by offering a channel for complaints. It is demonstrated when a reasoned challenge can still defeat the output and change the decision.

Footnotes

  1. High Court of Justice, Bates & Others v Post Office Limited (No. 6: Horizon Issues), [2019] EWHC 3408 (QB), especially paras. 905–913 and 963–1030. The findings concern the versions and periods examined in the litigation; the court expressly cautioned against applying them wholesale to Horizon as it stood in December 2019. ↩ ↩2

  2. Court of Appeal (Criminal Division), Hamilton & Others v Post Office Limited, [2021] EWCA Crim 577, especially paras. 1–5 and 447. The court allowed 39 appeals on both grounds and dismissed three. ↩

  3. Elham Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), National Institute of Standards and Technology, 2023, especially MEASURE 1.3 and 3.3 and MANAGE 2.4, 4.1–4.3. ↩

Oliver Wrede
Oliver Wrede

Oliver Wrede writes and teaches on interface design, knowledge systems, and the architecture of intelligence in organizations. He is interested in how humans, institutions, and machines reason together — and how design shapes the quality of that reasoning.

More from Oliver Wrede