DeVLBert
Authors:
Open Access
- 12 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4373-4382
Abstract
In this paper, we propose to investigate the problem of out-of-domain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned. Existing methods for this problem are purely likelihood-based, leading to the spurious correlations and hurt the generalization ability when transferred to out-of-domain downstream tasks. By spurious correlation, we mean that the conditional probability of one token (object or word) given another one can be high (due to the dataset biases) without robust (causal) relationships between them. To mitigate such dataset biases, we propose a Deconfounded Visio-Linguistic Bert framework, abbreviated as DeVLBert, to perform intervention-based learning. We borrow the idea of the backdoor adjustment from the research field of causality and propose several neural-network based architectures for Bert-style out-of-domain pretraining. The quantitative results on three downstream tasks, Image Retrieval (IR), Zero-shot IR, and Visual Question Answering, show the effectiveness of DeVLBert by boosting generalization ability.Keywords
All Related Versions
Funding Information
- Fundamental Research Funds for the Central Universities (2020QNA5024)
- Natural Science Foundation of Zhejiang Province (LR19F020006)
- National Natural Science Foundation of China (61625107, 61751209, 61836002)
- National Key Research and Development Program of China (2018AAA0101900, 2018AAA0100603)
This publication has 22 references indexed in Scilit:
- Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention2017
- Discovering Causal Signals in Images2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Learning Deep Structure-Preserving Image-Text Embeddings2016
- Deep Residual Learning for Image Recognition2016
- Creating Causal Embeddings for Question Answering with Minimal Supervision2016
- Exploring Causal Relationships in Visual Object Tracking2015
- VQA: Visual Question Answering2015
- NLTK2006
- CAUSALITY: MODELS, REASONING, AND INFERENCE, by Judea Pearl, Cambridge University Press, 2000Econometric Theory, 2003