Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
Open Access
- 28 September 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 7 (CSCW2) , 1-22
Abstract
Artificial intelligence (AI) is increasingly being considered to assist human decision-making in high-stake domains (e.g. health). However, researchers have discussed an issue that humans can over-rely on wrong suggestions of the AI model instead of achieving human AI complementary performance. In this work, we utilized salient feature explanations along with what-if, counterfactual explanations to make humans review AI suggestions more analytically to reduce overreliance on AI and explored the effect of these explanations on trust and reliance on AI during clinical decision-making. We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion, and analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our results showed that the AI model with both salient features and counterfactual explanations assisted therapists and laypersons to improve their performance and agreement level on the task when 'right' AI outputs are presented. While both therapists and laypersons over-relied on 'wrong' AI outputs, counterfactual explanations assisted both therapists and laypersons to reduce their over-reliance on 'wrong' AI outputs by 21% compared to salient feature explanations. Specifically, laypersons had higher performance degrades by 18.0 f1-score with salient feature explanations and 14.0 f1-score with counterfactual explanations than therapists with performance degrades of 8.6 and 2.8 f1-scores respectively. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on 'wrong' AI outputs and implications for improving human-AI collaborative decision-making.Keywords
All Related Versions
Funding Information
- The Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant
This publication has 45 references indexed in Scilit:
- Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic RetinopathyOphthalmology, 2018
- Unintended Consequences of Machine Learning in MedicineJAMA, 2017
- The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems2015
- Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelThe Annals of Applied Statistics, 2015
- Principles of Explanatory Debugging to Personalize Interactive Machine Learning2015
- Fugl-Meyer Assessment of Sensorimotor Function After StrokeStroke, 2011
- A review of instance selection methodsArtificial Intelligence Review, 2010
- Parkinson's disease: clinical features and diagnosisJournal of Neurology, Neurosurgery & Psychiatry, 2008
- Disagreement in Patient and Carer Assessment of Functional Abilities After StrokeStroke, 1999
- Treatment Techniques and Clinical Guidelines for Photocoagulation of Diabetic Macular EdemaOphthalmology, 1987