Did the Model Understand the Question?
Preprint
Abstract
We analyze state-of-the-art deep learning models for three tasks: question answering on (1) images, (2) tables, and (3) passages of text. Using the notion of \emph{attribution} (word importance), we find that these deep networks often ignore important question terms. Leveraging such behavior, we perturb questions to craft a variety of adversarial examples. Our strongest attacks drop the accuracy of a visual question answering model from $61.1\%$ to $19\%$, and that of a tabular question answering model from $33.5\%$ to $3.3\%$. Additionally, we show how attributions can strengthen attacks proposed by Jia and Liang (2017) on paragraph comprehension models. Our results demonstrate that attributions can augment standard measures of accuracy and empower investigation of model performance. When a model is accurate but for the wrong reasons, attributions can surface erroneous logic in the model that indicates inadequacies in the test data.
Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- A Novel Framework for Robustness Analysis of Visual QA ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering2017
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers2016
- Analyzing the Behavior of Visual Question Answering Models2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Inferring Logical Forms From Denotations2016