AI gone astray: How subtle shifts in patient data send popular algorithms reeling, undermining patient safety

Feb. 28, 2022

Data analysis by Adam Yala, Janice Yang, and Ludvig Karstens — Jameel Clinic, Massachusetts Institute of Technology

Mike Reddy for STAT

A novel investigation by STAT and the Massachusetts Institute of Technology found that subtle shifts in data fed into popular health care algorithms — used to warn caregivers of impending medical crises — can cause their accuracy to plummet over time, raising the prospect AI could do more harm than good in many hospitals.

In a monthslong experiment, STAT and MIT traced the performance of algorithms past their early days of peak performance into the grinding years that follow, when the hype has faded and they must prove their reliability to caregivers. Instead of transforming care, the algorithms withered in the face of fast-moving clinical conditions — unable to keep up with the pace of change.

Their frailty exposes gaping holes in the governance of products whose quiet deterioration in hospitals around the country threatens to mislead doctors and undermine patient safety. The initial signs of dysfunction are often faint, making it difficult to root out faulty information before it bleeds into decision-making.

STAT+ Exclusive Story