Better Understanding Differences in Attribution Methods via Systematic Evaluations
- 12 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (6) , 4090-4101
Abstract
Deep neural networks are very successful on many vision tasks, but hard to interpret due to their black box nature. To overcome this, various post-hoc attribution methods have been proposed to identify image regions most influential to the models’ decisions. Evaluating such methods is challenging since no ground truth attributions exist. We thus propose three novel evaluation schemes to more reliably measure the faithfulness of those methods, to make comparisons between them more fair , and to make visual inspection more systematic . To address faithfulness, we propose a novel evaluation setting (DiFull) in which we carefully control which parts of the input can influence the output in order to distinguish possible from impossible attributions. To address fairness, we note that different methods are applied at different layers, which skews any comparison, and so evaluate all methods on the same layers (ML-Att) and discuss how this impacts their performance on quantitative metrics. For more systematic visualizations, we propose a scheme (AggAtt) to qualitatively evaluate the methods on complete datasets. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models. Finally, we propose a post-processing smoothing step that significantly improves the performance of some attribution methods, and discuss its applicability.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- There and Back Again: Revisiting Backpropagation Saliency Methods2020
- Layer-Wise Relevance Propagation: An Overview2019
- Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation2019
- Top-Down Neural Attention by Excitation BackpropInternational Journal of Computer Vision, 2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- Explaining nonlinear classification decisions with deep Taylor decompositionPattern Recognition, 2017
- Deep Residual Learning for Image Recognition2016
- Learning Deep Features for Discriminative Localization2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Visualizing and Understanding Convolutional Networks2014