Risk for the Record

IBM, which sells software for governing AI, studied 600 organisations that had suffered a data breach in the year to February 2025. Of those, 63% had no policy for governing AI or were still writing one, and where AI models or applications had been breached, 97% of the organisations had no access controls on them.¹ So the first question about an AI risk conversation is whether there is one. Where there is one, it can exist mainly so that the project is seen to have handled risk.

Some of the risks on the register deserve their place there. But the people who choose which ones go on it are rewarded for looking covered, and the failures that break AI systems in production don’t make a good slide.

That makes the register a box ticker’s output in Graeber’s sense, a file that lets the organisation say it has handled something it hasn’t. An agent can now write all of it, the register and the quarterly pack, in an afternoon, and none of it gets more accurate for being cheap. The people who pay are the ones the failures happen to, and they never see the register.

When AI enters a workflow, the risks that matter are concrete and largely operational. Data leaves where it wasn’t meant to, because prompts and logs create paths the existing classification scheme didn’t anticipate.

Drift creates its own category. A vendor updates the model without asking, and a prompt that produced acceptable output last month produces subtly different output this month. Logging and monitoring designed to catch outages miss quality degradation, because quality degradation doesn’t look like an outage. Plausible-but-wrong output looks like a correct answer to a monitor built to catch malformed ones, so the dashboard stays green while the answers get worse.

Then there’s the category built by accident. A team installs a wrapper that makes the system easier to integrate and easier to misuse at the same time, and an adjacent department copies the pattern into a workflow with a different risk profile.

Stopping the System

A risk register says where harm could come from, and nothing about who can act on it in time.

The team that built the system is the worst placed to find its failure modes. They know what it was meant to do, and they have a release date to meet. Someone has to spend dedicated time trying to make the system do what the specification prohibits, and their findings have to go into the backlog with the weight of a customer-reported defect.

The kill switch and the rollback path need rehearsing before they’re needed. A switch that has never been pulled is a box on an architecture diagram. For an assistant in live use, pulling it out of production has to take minutes, and a change-control meeting next Tuesday is too late, however well it is minuted. The people who can authorise the pull need names, and an escalation path short enough to act inside the harm window.

And one person has to own it. They can stop a release, and they’re senior enough that saying no won’t end their career. Having somebody who could stop a release in principle isn’t the same as having somebody who can do it the week before launch and still be invited to the next steering committee. Without that person, the authority to halt doesn’t exist, and the threat model and red-team reports become documents the organisation will cite after the failure. It’s the independent auditor role, with the speed and the standing to use it.

A risk function has a quiet political-economy problem. It gets rewarded for the absence of visible incidents, and raising inconvenient exposures doesn’t pay in the same currency. A register with many risks implies that many things could go wrong, which reads as a function that isn’t fully in control. A function that lists few risks and reports them all under control gets its budget without friction. The selection pressure runs towards the short, green register.

And regulation makes the pattern worse. A risk conversation organised around compliance artefacts produces them at the frequency the regulator expects, whether or not the system they describe is still the system in production.

Diane Vaughan’s study of the Challenger launch found that the engineers’ concern about the O-rings had been raised and accepted, repeatedly, over years, until an anomaly that had once been a stop condition had become a known characteristic of the system. She called it the normalisation of deviance, and the mechanism was the paperwork: each review that accepted the anomaly made the next acceptance easier to file.² When a model’s error rate has been in the register for four quarters, it’s on the same path.

Committee-driven risk registers add their own mechanism. A register reviewed quarterly by a committee that doesn’t see the system in motion produces a snapshot of risk at the moment of the meeting, and the system changes between meetings while the snapshot decays. The register continues to be maintained, because the maintenance demonstrates a disciplined process, and a disciplined process is what the committee came to see. When an incident happens that the register didn’t list, the post-mortem adds it to the register for next quarter.

A captured risk function is not an idle one. It is still busy, with a full calendar and a quarterly pack, but the work has drifted away from the system it was meant to describe. A risk function that avoids capture reports findings the sponsors find inconvenient, and its head’s career doesn’t depend on keeping a sponsor happy. Without that, the register records when the committee met.

Fines for the Missing Form

The exposures above are paid for by other people first. The customer acts on the wrong answer in a generated letter, and the people in the records that left through a prompt find out later or never. The company that caused each pays a fraction, if anything, and nobody puts a cost it doesn’t bear on its own risk register, because nobody is paid to. So there is a sound case for a regulator. The harder question is what the regulator does next.

What a regulator can reach for is a short list: a rule, a record that the rule was followed, a penalty for the missing record, a profession to produce the records, and a market to check them. The AI Act is nothing if not thorough, and uses every item. The provider assigns each system a risk class. For a high-risk system the provider writes the conformity assessment and the technical documentation and keeps the logs, and the deployer appoints an overseer and, if it is a public body or a lender, files an impact assessment before use. Breaking the high-risk duties can cost up to 3% of global turnover, and the Act keeps its 7% cap for the practices it bans outright.³ Each duty comes with a familiar artefact, and it gets produced when the regulator’s calendar asks, which is not necessarily when the system changes. The fine punishes the missing document. Nobody fines the drifting system, and the organisation ends up keeping two sets of paperwork, the one Vaughan described and the one the inspector reads.

Europe has been through the whole sequence once before, with data. The General Data Protection Regulation has applied since May 2018.

What GDPR produced
Largest single fines	Meta, €1.2bn (2023); Amazon, €746m (2021, annulled on procedural grounds in 2026)⁴
Fines on record	2,685, to 1 March 2026⁵
Consent banners	on 62% of the most popular sites in the EU by October 2018 (required by an older cookie rule, on GDPR’s terms of consent)⁶

Now look at what became of the officer role. The law requires one, so organisations appoint one, sometimes by contract with an outside provider,⁷ seldom bring them in at the start,⁸ and give them no power to halt anything.⁹ Which makes the officer, in Graeber’s terms, a box ticker with a service agreement. In 2026 a Luxembourg court threw out the second-largest fine because the regulator had skipped a step in the proceedings, leaving its findings on the facts standing. So the facts stand, and the fine doesn’t. A compliance sector grew up to produce the records that show the data flows to be lawful. The AI Act’s officer, once there is one, is set up for the same treatment.

The consent banner teaches the worse lesson, because it shows what a rule built on paperwork does to the people it was meant to protect. Open a recipe for chocolate biscuits and the site puts a panel in front of it. An older EU rule on cookies requires the panel, and GDPR decides what counts as consent on it. Click accept, and the site and the 1,671 partners it calls trusted may store what you do there and share it across every site they carry. Refuse, and you work through the partner list one switch at a time, or go without the recipe. A vendor sells the site that banner by subscription, and an industry body keeps the partner list so that one click covers all of them. The only party to the panel who isn’t paid is the one who wanted the biscuits.

The regulation asks the site to process only what it needs, for the purpose it stated, and to show that it did. The reader never sees that duty. The reader sees the button, and the click turns whatever follows into something they agreed to, while every company on the list files a record that they did. The rule handed the work of understanding the processing to the person who wanted a recipe, which is the arrangement Graeber described between the rule-makers and the ruled, now with a button. GDPR does let a person claim damages, one claim at a time, with the burden of showing the harm on them, which leaves the work where the click put it. Make the site answer for harm done with the data, as a manufacturer answers for a product, and it can’t ask anyone to click that away. A deployer can satisfy the AI Act’s oversight article the same way: show a person the output, and have them tick the box.

Of course, penalties also create jobs. A fine large enough to matter hires lawyers on both sides, and funds the firms that audit prompts for a fee and sell readiness ahead of deadlines that keep moving, which is convenient, because readiness for a moved deadline can be sold twice. These are Graeber’s goons in the exact sense, people employed because the other side employs the equivalent, and the expected value of the fine pays for all of them. None of them shrinks the exposure.

The AI Act does ask a deployer for an overseer with “the necessary competence, training and authority”.¹⁰ It leaves the deployer to decide what that authority amounts to and whom the overseer answers to, and it says nothing about what happens to them after they use it. So an appointment letter can satisfy the article. The Union had drafted a second instrument, a directive that would have made it easier to claim damages for harm done by AI, and withdrew it in 2025 for lack of agreement.¹¹ The documents stayed. A revised product-liability law brings software, AI included, under the manufacturer’s rule for products sold from December 2026, and it will be the first test of whether the harm route does better.¹²

The part of a risk function that changes anything is small: someone paid to make the system do what the specification forbids, and someone senior enough to pull it out of production in minutes. Neither produces much paper. An agent can take on a good share of the first, attacking the system all night, which is a better use of one than writing the register.

Checklist

Sort your risk register: which items name a failure mode, an owner and a budget line, and which are vague?
Does the register describe the system in production, or the one approved last quarter? When did a change to a model, prompt or tool last update it?
Have you covered the ordinary exposures: data leaving through retrieval, prompts, logs and telemetry; vendors updating the model; uses you didn’t know you were building?
Does your risk function report findings the sponsors dislike, and does its head’s career depend on something other than the sponsor’s goodwill?
Who is paid to make your AI systems do what their specifications forbid, and who can pull one out of production in minutes?

Notes

IBM 2025.Source ↩
Vaughan 1996.↩
European Parliament and Council 2024, Art. 99.Source ↩
Data Protection Commission (Ireland) 2023. PPC Land 2026.Source Source ↩
CMS 2026.Source ↩
Degeling et al. 2019.Source ↩
IAPP 2019.Source ↩
Integritetsskyddsmyndigheten (IMY) 2023.Source ↩
European Parliament and Council 2016, Art. 39.Source ↩
European Parliament and Council 2024, Art. 26(2).Source ↩
European Commission 2025.Source ↩
European Parliament and Council 2024, Art. 2(1).Source ↩