Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Top Cited Papers
- 8 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (10) , 11624-11641
Abstract
Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named C ross- M odal C ausal Relat I onal R easoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering. The datasets, code, and models are available at https://github.com/HCPLab-SYSU/CMCIR.Keywords
Funding Information
- National Key R&D Program of China (2021ZD0111601)
- National Natural Science Foundation of China (62002395, 61976250)
- Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515011530, 2021A1515012311, 2020B1515020048)
- Guangzhou Science and Technology planning (2023A04J2030)
This publication has 72 references indexed in Scilit:
- Video Question Answering via Gradually Refined Attention over Appearance and Motion2017
- Visual Dialog2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering2017
- Deep Residual Learning for Image Recognition2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Stacked Attention Networks for Image Question Answering2016
- VQA: Visual Question Answering2015
- Glove: Global Vectors for Word Representation2014