Suboptimal Shapley Value Explanations
Author:
Preprint
Abstract
Deep Neural Networks (DNNs) have demonstrated strong capacity in supporting a wide variety of applications. Shapley value has emerged as a prominent tool to analyze feature importance to help people understand the inference process of deep neural models. Computing Shapley value function requires choosing a baseline to represent feature's missingness. However, existing random and conditional baselines could negatively influence the explanation. In this paper, by analyzing the suboptimality of different baselines, we identify the problematic baseline where the asymmetric interaction between $\bm{x}'_i$ (the replacement of the faithful influential feature) and other features has significant directional bias toward the model's output, and conclude that $p(y|\bm{x}'_i) = p(y)$ potentially minimizes the asymmetric interaction involving $\bm{x}'_i$. We further generalize the uninformativeness of $\bm{x}'_i$ toward the label space $L$ to avoid estimating $p(y)$ and design a simple uncertainty-based reweighting mechanism to accelerate the computation process. We conduct experiments on various NLP tasks and our quantitative analysis demonstrates the effectiveness of the proposed uncertainty-based reweighting mechanism. Furthermore, by measuring the consistency of explanations generated by explainable methods and human, we highlight the disparity between model inference and human understanding.
Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- WeightedSHAP: Analyzing and Improving Shapley Based Feature Attributions2022
- Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importanceStatistics and Computing, 2021
- ERASER: A Benchmark to Evaluate Rationalized NLP Models2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Explaining prediction models and individual predictions with feature contributionsKnowledge and Information Systems, 2013