Every AI vendor will show you an accuracy number. 94.7%. 97.2%. Sometimes, if they're feeling bold, 99.1%. These numbers are real. They're also, in the context of actual deployment, almost completely beside the point.

Here's the problem. Accuracy is a mean. And means hide tails.

Nassim Taleb made this point mercilessly in Antifragile and Skin in the Game: a strategy that accumulates many small gains can be entirely rational on average and still be ruinous in practice — if the downside is unbounded. You can win ninety-nine times and be wiped out on the hundredth. The issue isn't your average. The issue is what happens when you're wrong in a way that really matters.

AI in production has exactly this structure. Most decisions are routine. The model handles them correctly, value accrues quietly, no one notices. Then comes the edge case — the invoice with a fraudulent vendor, the incident ticket that looked like a billing query but was actually a security breach, the customer reply that promised a refund the company can't honour. If your system auto-committed on all of those with equal confidence, you've just let the model's tail risk become your operational tail risk.

The Decision Rule That Caps the Tail

The right frame isn't "how often is the model correct?" It's "when the model is wrong, how bad can it get?" And crucially: "how do I prevent the worst outcomes from happening automatically?"

The answer is a decision rule: only take autonomous action when p < C/L.

Where:

  • p is the calibrated probability of error on this specific output — not the benchmark average, but the actual uncertainty on this decision, right now
  • L is the loss if the model is wrong and the action commits
  • C is the cost of caution — routing to a human, holding for review, or simply not acting

Rearranged: you act autonomously only when the expected loss (p × L) is less than the cost of doing nothing. The moment that ratio flips, you escalate. It's not a vibe. It's arithmetic.

Three Decisions, Same Model, Different Stakes

The power of this rule is that it responds to context, not just confidence. Consider three decisions a model might face in a single afternoon:

Invoice approval — L = €1,500

A supplier invoice comes in. The model reads it, matches it to a purchase order, and returns a confidence score. Your cost of caution — the loaded time cost of a finance team member reviewing it — is around €8. The loss if you approve a fraudulent invoice: €1,500.

At 0.5% error probability: expected loss = €7.50. Cost of caution = €8. Auto-approve — caution costs more than the expected risk.

Incident triage — L = €50,000

An alert comes in flagged as a Severity 3 network anomaly. Same confidence score. But now the loss if misrouted is €50,000 — delayed response, customer SLA breach, potential data exposure.

At 0.5% error probability: expected loss = €250. Cost of caution (engineer review) = €80. Escalate — the expected loss is three times the cost of a human decision.

Same model confidence. Completely different action. Because L is different.

Support reply — L = €100

A customer asks a standard billing question. The model drafts a reply. L = €100 — the cost of a wrong answer requiring a follow-up call. C = €2 for a thirty-second agent review.

At 0.5% error probability: expected loss = €0.50. Cost of caution = €2. Auto-send — caution costs four times the risk.

But now imagine the model's uncertainty spikes — it's been asked something touching a recent policy change. Error probability jumps to 5%. Expected loss = €5. Cost of caution = €2. The rule flips: escalate.

Accuracy Is an Average. Safety Is a Guarantee About the Tail.

This is the distinction that matters. A 97% accurate model used naively on high-L decisions will, at scale, produce catastrophic outcomes roughly 3% of the time — which at a thousand decisions a day is thirty disasters. A model operating under a principled decision rule produces far fewer autonomous actions on high-L cases, and routes the uncertain ones to humans precisely when the economics justify the interruption.

Taleb's insight about bounded downside applies directly. The decision rule doesn't improve your average. It caps your tail. You give up a small amount of automation throughput in exchange for a structural guarantee that no single bad output can become a ruinous loss.

The benchmark number tells you how the model performs on average. The decision rule tells you what happens when it doesn't. Getting that second question right is the difference between a useful AI deployment and a liability.

Most teams optimise the first number. The first team that optimises the second one will be the one their board actually trusts.