Imperfections of XAI: Phenomena Influencing AI-Assisted Decision-Making
Open Access
- 9 September 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Interactive Intelligent Systems
- Vol. 15 (3) , 1-40
Abstract
With the increasing use of AI, recent research in human–computer interaction explores Explainable AI (XAI) to make AI advice more interpretable. While research addresses the effects of incorrect AI advice on AI-assisted decision-making, the impact of incorrect explanations is neglected so far. Additionally, recent work shows that not only different explanation modalities impact decision-makers, but also human factors play a critical role. To analyze relevant phenomena influencing AI-assisted decision-making, this work explores the impacting factors by conceptualizing theories of appropriate reliance and taking the first steps toward empirical evidence. We show that humans’ reliance on AI and the human–AI team performance are impacted by imperfect XAI in a study with 136 participants. Additionally, we find that cognitive styles affect decision-making in different explanation modalities. Hence, we shed light on diverse factors that impact human–AI collaboration and provide guidelines for designers to tailor such human–AI collaboration systems to individuals’ needs.Keywords
Funding Information
- National Heart, Lung, and Blood Institute of the National Institutes of Health (R01HL164906)
This publication has 73 references indexed in Scilit:
- Assessing fidelity in XAI post-hoc techniques: A comparative study with ground truth explanations datasetsArtificial Intelligence, 2024
- GitHub Copilot AI pair programmer: Asset or Liability?Journal of Systems and Software, 2023
- How should the results of artificial intelligence be explained to users? - Research on consumer preferences in user-centered explainable artificial intelligenceTechnological Forecasting and Social Change, 2023
- The perils and pitfalls of explainable AI: Strategies for explaining algorithmic decision-makingGovernment Information Quarterly, 2022
- The effects of domain knowledge on trust in explainable AI and task performance: A case of peer-to-peer lendingInternational Journal of Human-Computer Studies, 2022
- Generating visual explanations with natural languageApplied AI Letters, 2021
- Human–computer collaboration for skin cancer recognitionNature Medicine, 2020
- Egoistic and altruistic motivation: How to induce users’ willingness to help for imperfect AIComputers in Human Behavior, 2019
- Prolific.ac—A subject pool for online experimentsJournal of Behavioral and Experimental Finance, 2018
- Beyond the Turk: Alternative platforms for crowdsourcing behavioral researchJournal of Experimental Social Psychology, 2017