Counterfactual Vision and Language Learning
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10041-10051
Abstract
The ongoing success of visual question answering methods has been somwehat surprising given that, at its most general, the problem requires understanding the entire variety of both visual and language stimuli. It is particularly remarkable that this success has been achieved on the basis of comparatively small datasets, given the scale of the problem. One explanation is that this has been accomplished partly by exploiting bias in the datasets rather than developing deeper multi-modal reasoning. This fundamentally limits the generalization of the method, and thus its practical applicability. We propose a method that addresses this problem by introducing counterfactuals in the training. In doing so we leverage structural causal models for counterfactual evaluation to formulate alternatives, for instance, questions that could be asked of the same image set. We show that simulating plausible alternative training data through this process results in better generalization.Keywords
This publication has 18 references indexed in Scilit:
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017
- Infinite Variational Autoencoder for Semi-Supervised Learning2017
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- Deep Residual Learning for Image Recognition2016
- Glove: Global Vectors for Word Representation2014
- Causality2009
- The Nonstochastic Multiarmed Bandit ProblemSIAM Journal on Computing, 2002