Aviation solved the vigilance problem. AI just gave security a worse one

Contributor

An air traffic controller watching a busy scope will, sooner or later, miss the one aircraft that matters. Sustained attention decays under load, a limit aviation named the vigilance decrement and has spent seventy years designing around.

AI has moved every knowledge worker into that chair. You now work a dozen aircraft at once: six open conversations, an agent drafting in the background, three more waiting, and someone calls it efficiency. Your scope is your inbox. The vigilance decrement does not care that you were told to keep up.

This is where aviation’s rule actually lands, and it lands on the dumb machine. Guard those actions with something that fires on the event itself, not on a human’s read of it: a mandatory callback to a known number for any change of bank details, dual control on a wire, a hard threshold above which a payment stops on its own.

None of this is new. These controls used to serve as belt and suspenders behind an email filter that caught the obvious lure. The tell is disappearing, so these controls have to carry more of the load. Like the collision system, they read no one’s intent. They fire on the act, every time, and ask no one to be clever.

What they miss is the wire that is already coherent: a request from a genuinely compromised vendor thread, correct in every detail, that a callback confirms because the fraudster holds the other end too. Catching that means judging whether an action makes sense in context, the adversarial problem a collision system never had to solve.

This is where aviation’s rule stops transferring, at least for now. You cannot tell anyone to follow this machine because it is not the dumb, certain kind. It guesses in an adversarial fog, and a wrong guess halts a real wire. Any machine in the money path earns its place only if it is quiet enough to be trusted and cheap enough when wrong that treasury does not route around it.

A noisy machine just relocates the vigilance decrement onto the backup, who clears its alerts on the same schedule the controller missed the blips. The reliability required to ever say “follow the machine” is the problem security still has to solve.

The question worth ending on

Aviation reformed because the cost of pretending was a wreck in a field with 71 names on it. Security’s cost is quieter: a wire that clears and an identity handed over, so we keep asking people to do the one thing aviation proved they cannot: serve as the reliable last line against a rare event buried in routine. AI has made that bet even worse.

The discipline that learned this first did not demand more vigilance. It built controls around the moments where failure carried the greatest consequence and stopped depending on an overloaded person to catch every dangerous event.

Security already has its version of the dumb machine: controls that fire on the act, regardless of whether anyone recognizes the attack. The machine that reads intent has to earn that same trust before anyone follows it.

So the question is not whether your people are paying attention. Aviation could have told you decades ago that, eventually, they will not. The question is whether, at the moment someone is about to move money or surrender identity, anything stands in the path except a human hoping to notice.

Contributor