A recurring theme across documented AI harms isn’t that the system was wrong — every system is sometimes wrong. It’s that a human in the loop deferred to it anyway. The research term for this is automation bias: the tendency to trust an automated output over your own judgement, especially under time pressure.

It shows up everywhere the record has been examined. An officer trusts a facial match they should have questioned. A caseworker accepts a risk score they don’t understand. A driver assumes the system is handling something it isn’t.

This matters because the usual fix — “keep a human in the loop” — quietly assumes the human will overrule a bad output. Often they don’t. A human who rubber-stamps the machine is not a safeguard; they’re a liability shield. Any honest account of AI risk has to include the people, not just the models.