Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing
Authors:
Preprint
Abstract
Despite significant success in Visual Question Answering (VQA), VQA models have been shown to be notoriously brittle to linguistic variations in the questions. Due to deficiencies in models and datasets, today's models often rely on correlations rather than predictions that are causal w.r.t. data. In this paper, we propose a novel way to analyze and measure the robustness of the state of the art models w.r.t semantic visual variations as well as propose ways to make models more robust against spurious correlations. Our method performs automated semantic image manipulations and tests for consistency in model predictions to quantify the model robustness as well as generate synthetic data to counter these problems. We perform our analysis on three diverse, state of the art VQA models and diverse question types with a particular focus on challenging counting questions. In addition, we show that models can be made significantly more robust against inconsistent predictions using our edited data. Finally, we show that results also translate to real-world error cases of state of the art models, which results in improved overall performanceKeywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Explainable neural computation via stack neural module networksApplied AI Letters, 2021
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering2018
- Visual question answering: Datasets, algorithms, and future challengesComputer Vision and Image Understanding, 2017
- Visual question answering: A survey of methods and datasetsComputer Vision and Image Understanding, 2017
- Deep Residual Learning for Image Recognition2016
- Yin and Yang: Balancing and Answering Binary Visual Questions2016
- Stacked Attention Networks for Image Question Answering2016
- Microsoft COCO: Common Objects in Context2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998