知Yizhi Daily
← Back to this edition

AI

Study Shows Doctors Struggle to Detect Failures in AI Diagnostic Systems in Time

Edition 2026-11-16 · 08:00 Beijing time

In a randomized controlled study involving 48 medical students and doctors, participants used AI with progressively declining reliability for cardiac auscultation diagnoses, with the AI's accuracy dropping from 90% to 0%. Only 12.5% noticed that the system had been deliberately degraded; even when diagnosing independently before viewing AI recommendations, diagnostic performance still declined as AI reliability fell, and doctors' confidence in their own diagnoses dropped faster than their trust in AI.

A study summary published by Reddit Science shows that researchers conducted a randomized controlled experiment around AI-assisted cardiac auscultation. Forty-eight doctors or medical students were assigned to either a "view AI first" condition or a "judge independently first, then view AI" condition. Each participant had to classify 40 digital heart sounds as normal or abnormal, and the experiment was conducted in four blocks.

Participants were told in advance that the AI had demonstrated 96% accuracy, 97% sensitivity, and 89% specificity during training and validation. To simulate system failure, the AI's reliability dropped across the four blocks to 90%, 60%, 50%, and 0%, respectively. The final block also deliberately mismatched auditory and visual cues to provide additional signals of abnormality.

The study examined whether humans could detect failure and correct erroneous output after AI had performed reliably for a long period. The paper notes that automation bias may lead doctors to treat AI recommendations as a shortcut replacing active verification, and AI output may also form a cognitive anchor that induces confirmation bias. The "cognitive forcing" strategy in the study design required doctors to classify independently before seeing AI recommendations, in order to prompt them to process clinical information.

The results showed that diagnostic performance still worsened as AI reliability declined, even when participants were required to make an initial classification first, and even when human performance exceeded AI reliability. The paper said only 12.5% of participants noticed that the AI diagnostic system's accuracy had been deliberately reduced; participants' confidence in their own diagnoses fell faster than their trust in AI. Based on this, researchers proposed that the goals and processes of human oversight should be clearly defined to protect patient safety.

Original sources

Reddit Science ↗
Back to this editionOpen interactive reader