Challenges for human oversight in detecting system malfunctions in AI-assisted medical diagnostics

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Increasingly, physicians must oversee AI systems to detect system malfunctions. We studied 48 physicians in a randomized controlled study using an increasingly malfunctioning AI for cardiac auscultation. Diagnostic performance declined alongside AI reliability, even when physicians were cognitively forced to make an initial classification, and when human performance exceeded AI reliability. Physicians’ confidence declined faster than their trust in AI. Results call for specification of oversight goals and processes ensuring patient safety.
Funding Information
  • Deutsche Forschungsgemeinschaft (389792660 as part of TRR 248 – CPEC)

This publication has 32 references indexed in Scilit: