DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
- 16 July 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 3369-3380
Abstract
Video question answering is a challenging task, which requires agents to be able to understand rich video contents and perform spatial-temporal reasoning. However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same video, different questions may require different amount of video clips or objects to infer the answer with relational reasoning; (2) During reasoning, appearance and motion features have complicated interdependence which are correlated and complementary to each other. Based on these observations, we propose a Dual-Visual Graph Reasoning Unit (DualVGR) which reasons over videos in an end-to-end fashion. The first contribution of our DualVGR is the design of an explainable Query Punishment Module, which can filter out irrelevant visual features through multiple cycles of reasoning. The second contribution is the proposed Video-based Multi-view Graph Attention Network, which captures the relations between appearance and motion features. Our DualVGR network achieves state-of-the-art performance on the benchmark MSVD-QA and SVQA datasets, and demonstrates competitive results on benchmark MSRVTT-QA datasets. Our code is available at https://github.com/MM-IR/DualVGR-VideoQA .Keywords
All Related Versions
Funding Information
- National Key Research and Development Plan of China (2020AAA0106200)
- National Natural Science Foundation of China (61936005, 62036012, 61872424)
- Natural Science Foundation of Jiangsu Province (BK20200037)
This publication has 43 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Stacked Attention Networks for Image Question Answering2016
- Neural Module Networks2016
- Bilinear CNN Models for Fine-Grained Visual Recognition2015
- VQA: Visual Question Answering2015
- Glove: Global Vectors for Word Representation2014
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Measuring Statistical Dependence with Hilbert-Schmidt Norms2005
- Separating Style and Content with Bilinear ModelsNeural Computation, 2000