How Stated Accuracy of an AI System and Analogies to Explain Accuracy Affect Human Reliance on the System
Open Access
- 28 September 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 7 (CSCW2) , 1-29
Abstract
AI systems are increasingly being used to support human decision making. It is important that AI advice is followed appropriately. However, according to existing literature, users typically under-rely or over-rely on AI systems, and this leads to sub-optimal team performance. In this context, we investigate the role of stated system accuracy by contrasting the lack of system information with the presence of system accuracy in a loan prediction task. We explore how the degree to which humans understand system accuracy influences their reliance on the AI system, by investigating numeracy levels and with the aid of analogies to explain system accuracy in a first of its kind between-subjects study (N=281). We found that explaining the stated accuracy of a system using analogies failed to help users rely on the AI systemappropriately (i.e., the tendency of users to rely on the system when the system is correct, or on themselves otherwise). To eliminate the impact of subjective attitudes towards analogy domains, we conducted a within-subjects study (N=248) where each participant worked on tasks with analogy-based explanations from different domains. Results from this second study confirmed that explaining stated accuracy of the system with analogies was not sufficient to facilitate appropriate reliance on the AI system in the context of loan prediction tasks, irrespective of individual user differences. Based on our findings from the two studies, we reason that the under-reliance on the AI system may be a result of users' overestimation of their own ability to solve the given task. Thus, although familiar analogies can be effective in improving the intelligibility of stated accuracy of the system, an improved understanding of system accuracy does not necessarily lead to improved system reliance and team performance.Keywords
This publication has 44 references indexed in Scilit:
- Evaluation of clustering algorithms for financial risk analysis using MCDM methodsInformation Sciences, 2014
- Using Comparison Scenarios to Improve Prenatal Risk CommunicationMedical Decision Making, 2012
- Using Analogies to Communicate Information about Health RisksApplied Cognitive Psychology, 2012
- From chance to choice: The use of a verbal analogy in the communication of riskHealth, Risk & Society, 2010
- Effect of Risk Ladder Format on Risk Perception in High‐ and Low‐Numerate IndividualsRisk Analysis, 2009
- Validation of the Subjective Numeracy Scale: Effects of Low Numeracy on Comprehension of Risk Communications and Utility ElicitationsMedical Decision Making, 2007
- Measuring Numeracy without a Math Test: Development of the Subjective Numeracy ScaleMedical Decision Making, 2007
- Behavioral Economics and Marketing in Aid of Decision Making among the PoorJournal of Public Policy & Marketing, 2006
- Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments.Journal of Personality and Social Psychology, 1999
- Teaching statistics to health professionals: The legal analogyMedical Teacher, 1992