Trust criteria for artificial intelligence in health: normative and epistemic considerations
- 18 November 2023
- journal article
- research article
- Published by BMJ in Journal of Medical Ethics
- Vol. 50 (8) , 544-551
Abstract
Rapid advancements in artificial intelligence and machine learning (AI/ML) in healthcare raise pressing questions about how much users should trust AI/ML systems, particularly for high stakes clinical decision-making. Ensuring that user trust is properly calibrated to a tool’s computational capacities and limitations has both practical and ethical implications, given that overtrust or undertrust can influence over-reliance or under-reliance on algorithmic tools, with significant implications for patient safety and health outcomes. It is, thus, important to better understand how variability in trust criteria across stakeholders, settings, tools and use cases may influence approaches to using AI/ML tools in real settings. As part of a 5-year, multi-institutional Agency for Health Care Research and Quality-funded study, we identify trust criteria for a survival prediction algorithm intended to support clinical decision-making for left ventricular assist device therapy, using semistructured interviews (n=40) with patients and physicians, analysed via thematic analysis. Findings suggest that physicians and patients share similar empirical considerations for trust, which were primarilyepistemicin nature, focused on accuracy and validity of AI/ML estimates. Trust evaluations considered the nature, integrity and relevance of training data rather than the computational nature of algorithms themselves, suggesting a need to distinguish ‘source’ from ‘functional’ explainability. To a lesser extent, trust criteria were also relational (endorsement from others) and sometimes based on personal beliefs and experience. We discuss implications for promoting appropriate and responsible trust calibration for clinical decision-making use AI/ML.Keywords
Funding Information
- Agency for Healthcare Research and Quality (R01HS027784)
This publication has 34 references indexed in Scilit:
- Development and validation of a patient-centered knowledge scale for left ventricular assist device placementThe Journal of Heart and Lung Transplantation, 2016
- The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems2015
- Assessment of patients’ and caregivers’ informational and decisional needs for left ventricular assist device placement: Implications for informed consent and shared decision-makingThe Journal of Heart and Lung Transplantation, 2015
- Trust in AutomationHuman Factors: The Journal of the Human Factors and Ergonomics Society, 2014
- Treatment of DystoniaNeurotherapeutics, 2014
- The Influence of Task Load and Automation Trust on Deception DetectionGroup Decision and Negotiation, 2004
- Trust in Automation: Designing for Appropriate RelianceHuman Factors: The Journal of the Human Factors and Ergonomics Society, 2004
- Persuasiveness of expert systemsBehaviour & Information Technology, 1998
- Trust in close relationships.Journal of Personality and Social Psychology, 1985
- A Behavioral Decision Theory Approach to Modeling Dyadic Trust in Superiors and SubordinatesPsychological Reports, 1984