Beyond the Black Box: Unraveling the Role of Explainability in Human-Artificial Intelligence Collaboration
- 19 August 2026
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
Abstract
Explainable artificial intelligence (AI) models have been proposed to mitigate overreliance and underreliance on AI, which reduce the effectiveness of human-AI collaborative tools. Yet, empirical evidence is mixed, and the impact of explainable AI on the cognitive effort and fatigue of a decision maker (DM) is often overlooked. This paper offers a theoretical perspective on these issues. We develop an analytical model that incorporates the defining features of human and machine intelligence, capturing the limited but flexible nature of human cognition with imperfect machine recommendations. Crucially, we represent how AI-based explanations influence the DM’s belief in the algorithm’s predictive quality. Our results indicate that explainable AI has varying effects depending on the level of explainability provided. Although low explainability levels have no impact on decision accuracy and reliance behavior, they lessen the cognitive burden on the DM. In contrast, higher explainability levels enhance accuracy by improving overreliance but at the expense of increased underreliance. Further, the relative impact of explainability is higher when the DM is more cognitively constrained, when the decision task is sufficiently complex, or when the stakes are lower. Importantly, higher explainability levels can escalate the DM’s cognitive burden (and hence, overall processing time and fatigue) precisely when explanations are most needed (i.e., when the DM is pressed for time to complete a complex task and doubts the machine’s quality). Our study clarifies how explainability affects decision outcomes and cognitive effort, informing the design of effective human-AI systems across decision environments. This paper was accepted by Jeannette Song, operations management. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.07450 .Keywords
This publication has 41 references indexed in Scilit:
- Algorithmic Decision‐Making Safeguarded by Human KnowledgeNaval Research Logistics (NRL), 2026
- Be careful what you explain: Benefits and costs of explainable AI in a simulated medical taskComputers in Human Behavior: Artificial Humans, 2023
- Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision taskComputers in Human Behavior, 2022
- Fatigue, personnel scheduling and operations: Review and research opportunitiesEuropean Journal of Operational Research, 2021
- Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in XAI user studiesArtificial Intelligence, 2021
- Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological imagesJournal of Biomedical Informatics, 2020
- Human–machine partnership with artificial intelligence for chest radiograph diagnosisnpj Digital Medicine, 2019
- A systematic review of algorithm aversion in augmented decision makingJournal of Behavioral Decision Making, 2019
- Making sense of recommendationsJournal of Behavioral Decision Making, 2019
- Implications of rational inattentionJournal of Monetary Economics, 2003