Causal Attention for Vision-Language Tasks
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9842-9852
Abstract
We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the QK-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT [57], which uses fewer data and less computational power, comparable to the heavier UNITER [14]. Code is published in https://github.com/yangxuntu/lxmertcatt.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- VQA: Visual Question Answering2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- External Validity: From Do-Calculus to Transportability Across PopulationsStatistical Science, 2014
- Causal Inference Using Potential OutcomesJournal of the American Statistical Association, 2005
- The Pareto, Zipf and other power lawsEconomics Letters, 2001
- The Dynamic Representation of ScenesVisual Cognition, 2000
- The Retinotopy of Visual Spatial AttentionNeuron, 1998
- Causal diagrams for empirical researchBiometrika, 1995