Guiding Audio-Visual Question Answering with Collective Question Reasoning
Authors:
Open Access
- 3 July 2025
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (10) , 6912-6929
Abstract
Audio-Visual Question Answering (AVQA) requires the model to answer questions with complex dynamic audio-visual information. Prior works on this task mainly consider only using single question-answer pairs during training, overlooking the rich semantic associations between questions. In this work, we propose a novel Collective Question-Guided Network (CoQo), which accepts multiple question-answer pairs as input and leverages the reasoning over these questions to assist the model training process. The core module is the proposed Question Guided Transformer (QGT), which uses collective question reasoning to perform question-guided feature extraction. Since multiple question-answer pairs are not always available, especially during inference, our QGT uses a set of learnable tokens to learn the collective information from multiple questions during training. At inference time, these learnable tokens bring additional reasoning information even when only one question is used as input. We employ QGT in both spatial and temporal dimensions to extract question-related features effectively and efficiently. To better capture detailed audio-visual associations, we train the model in a finer level by distinguishing feature pairs of different questions within the same video. Extensive experiments demonstrate that our method can achieve state-of-the-art performance on three AVQA datasets while reducing training time significantly. We also observe strong performances of our method on three VQA benchmarks. Detailed ablation studies further confirm the effectiveness of our proposed collective question reasoning scheme, both quantitatively and qualitatively.Keywords
Funding Information
- National Key R&D Program of China (2022ZD0160102)
- Shanghai Committee of Science and Technology (22YF1461500)
This publication has 64 references indexed in Scilit:
- Compound Prototype Matching for Few-Shot Action Recognition2022
- Dual Perspective Network for Audio-Visual Event Localization2022
- Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing2022
- Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing2020
- Multiple Sound Sources Localization from Coarse to Fine2020
- Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation2020
- Audio-Visual Event Localization in Unconstrained Videos2018
- Question-Guided Hybrid Convolution for Visual Question Answering2018
- The Sound of Pixels2018
- Predicting Gaze in Egocentric Video by Learning Task-Dependent Attention Transition2018