Capability Is Not Reliability

Start with the consequence of being wrong

AI can produce remarkable results and still fail often enough that you would not hand it an important job without supervision.

That is not a contradiction. Capability asks whether success is possible. Reliability asks whether success is consistent enough for the consequence attached to the work.

Research increasingly makes that gap visible. In a preregistered experiment with 758 BCG consultants, AI users completed 12.2% more within-frontier tasks and did them 25.1% faster on average, yet on a selected outside-frontier managerial task they were 19% less likely to be correct. METR’s task-horizon work separately shows that reliability falls as tasks become longer. One success does not define the frontier, and the frontier moves. Sources: BCG / Organization Science · METR

The reliability bar changes with the consequence

There is no universal number called “reliable enough.”

The standard changes with the task. A bad restaurant suggestion is cheap. Deleting the wrong files, sending the wrong message, changing production software, making a financial commitment, or acting on sensitive information carries a different cost.

Model accuracy is only part of operational reliability. Consequence, reversibility, detectability, verification, recovery, and authority all change the bar.

Unpredictability creates supervision work

A system that succeeds often but unpredictably can create a large supervision burden. The human has to decide which outputs deserve checking, catch failures that look plausible, and recover when the system crosses its own capability boundary.

That is one reason a powerful model can feel less useful in a real workflow than in a demo. The missing variable is not intelligence. It is dependable operation.

Trustworthy does not mean infallible

Useful technologies are trusted because important failure modes are understood, monitored, bounded, and recoverable.

For AI, that means making important things visible: what it did, what evidence supports the result, where it is uncertain, what authority it has, what can be reversed, and what happens when confidence is not good enough.

Trust is not believing something will never fail. Trust is knowing what failure means and what happens next.

Set the delegation scope from evidence and consequence

There is a bad version of safety where a person has to approve every click. There is also a bad version of autonomy where a system acts simply because it technically can.

A more useful approach is to scale delegation to consequence and evidence.

Routine, reversible work can happen with less interruption. As consequences rise, the system should need stronger evidence, clearer authority, and a credible recovery path before it acts.

Use consequence to set the bar

AI capability will keep improving.

That is useful progress.

But demonstrated capability is only one part of the decision.

For a real task, the decision is whether the measured reliability is high enough for the cost of being wrong.