Quantifying Explainability of Saliency Methods in Deep Neural Networks With a Synthetic Dataset
Open Access
- 13 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 4 (4) , 858-870
Abstract
Post-hoc analysis is a popular category in eXplainable artificial intelligence (XAI) study. In particular, methods that generate heatmaps have been used to explain the deep neural network (DNN), a black-box model. Heatmaps can be appealing due to the intuitive and visual ways to understand them but assessing their qualities might not be straightforward. Different ways to assess heatmaps' quality have their own merits and shortcomings. This paper introduces a synthetic dataset that can be generated adhoc along with the ground-truth heatmaps for more objective quantitative assessment. Each sample data is an image of a cell with easily recognized features that are distinguished from localization ground-truth mask, hence facilitating a more transparent assessment of different XAI methods. Comparison and recommendations are made, shortcomings are clarified along with suggestions for future research directions to handle the finer details of select post-hoc analysis methods. Furthermore, mabCAM is introduced as the heatmap generation method compatible with our ground-truth heatmaps. The framework is easily generalizable and uses only standard deep learning components.Keywords
All Related Versions
Funding Information
- Alibaba Group Holding Limited, DAMO Academy, Health-AI Division
- Alibaba-NTU Talent Program, Alibaba JRI
- Alibaba and Nanyang Technological University, Singapore
- RIE2020 AME Programmatic Fund, Singapore (A20G8b0102)
This publication has 33 references indexed in Scilit:
- Interpreting Deep Visual Representations via Network DissectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Interpretable Explanations of Black Boxes by Meaningful Perturbation2017
- Network Dissection: Quantifying Interpretability of Deep Visual Representations2017
- Evaluating the Visualization of What a Deep Neural Network Has LearnedIEEE Transactions on Neural Networks and Learning Systems, 2016
- "Why Should I Trust You?"2016
- Deep Residual Learning for Image Recognition2016
- Learning Deep Features for Discriminative Localization2016
- Very deep convolutional neural network based image classification using small training sample size2015
- On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance PropagationPLOS ONE, 2015
- Volume under the ROC Surface for Multi-class Problems2003