What Makes for a Good Saliency Map? Comparing Strategies for Evaluating Saliency Maps in Explainable AI (XAI)
Preprint
Abstract
Saliency maps are a popular approach for explaining classifications of (convolutional) neural networks. However, it remains an open question as to how best to evaluate salience maps, with three families of evaluation methods commonly being used: subjective user measures, objective user measures, and mathematical metrics. We examine three of the most popular saliency map approaches (viz., LIME, Grad-CAM, and Guided Backpropagation) in a between subject study (N=166) across these families of evaluation methods. We test 1) for subjective measures, if the maps differ with respect to user trust and satisfaction; 2) for objective measures, if the maps increase users' abilities and thus understanding of a model; 3) for mathematical metrics, which map achieves the best ratings across metrics; and 4) whether the mathematical metrics can be associated with objective user measures. To our knowledge, our study is the first to compare several salience maps across all these evaluation methods$-$with the finding that they do not agree in their assessment (i.e., there was no difference concerning trust and satisfaction, Grad-CAM improved users' abilities best, and Guided Backpropagation had the most favorable mathematical metrics). Additionally, we show that some mathematical metrics were associated with user understanding, although this relationship was often counterintuitive. We discuss these findings in light of general debates concerning the complementary use of user studies and mathematical metrics in the evaluation of explainable AI (XAI) approaches.Keywords
All Related Versions
This publication has 64 references indexed in Scilit:
- Conceptualizing understanding in explainable artificial intelligence (XAI): an abilities-based approachEthics and Information Technology, 2024
- HIVE: Evaluating the Human Interpretability of Visual Explanations2022
- Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyondKnowledge and Information Systems, 2022
- Metrics for Saliency Map Evaluation of Deep Learning Explanation Methods2022
- From Responsibility to Reason-Giving Explainable Artificial IntelligencePhilosophy & Technology, 2022
- What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI researchArtificial Intelligence, 2021
- The automation of leadership functions: Would people trust decision algorithms?Computers in Human Behavior, 2021
- Artificial intelligence in retail: The AI-enabled value chainAustralasian Marketing Journal, 2020
- Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015