Unbiased Visual Question Answering by Leveraging Instrumental Variable
- 26 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 6648-6662
Abstract
Existing unbiased VQA models reduce the spurious correlation between questions and answers to force the models to focus on visual information. However, the visual information captured by these unbiased models is irrelevant to the correct answer, resulting in leveraging spurious correlation to predict incorrect answers. This makes these unbiased methods fail to obtain critical visual information, thus performing poorly on questions dominated by the visual information. To capture the valuable visual information, this paper proposes a novel unbiased VQA model based on causal inference, leveraging Instrumental Variable (IVar) to increase the causal effect between visual features and answers. First, to obtain suitable instrumental variables, the noise generator is proposed according to the constraints of IVar. The generated noise can be regarded as IVar, which is used to pollute the original visual features. Then, this paper proposes IVar loss which utilizes the generated IVar to increase the causal effect between visual features and answers. When the visual feature is polluted by IVar, IVar loss guides the model to predict incorrect answers to enhance the correlation between IVar and the answer. Since the correlation between IVar and the answer is proportional to the causal effect between the visual feature and the answer, IVar loss enhances the importance of the visual information, thereby rectifying the model to capture critical visual information. The extensive experimental results on widely-used benchmarks demonstrate the advantages of the proposed method. The proposed method gains the best accuracy on answer type Other of VQA-CP v2. These results demonstrate the superiority of the proposed method in capturing critical visual information since most questions on the answer type Other are dominated by visual information.Keywords
Funding Information
- National Key Research and Development Program of China (2022ZD0118802)
- National Natural Science Foundation of China (U20B2064, 62372233)
This publication has 39 references indexed in Scilit:
- Rethinking Data Augmentation for Robust Visual Question Answering2022
- Towards Non-I.I.D. image classification: A dataset and baselinesPattern Recognition, 2021
- Unbiased Scene Graph Generation From Biased Training2020
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- From Recognition to Cognition: Visual Commonsense Reasoning2019
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Discovering Causal Signals in Images2017
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017