Modality-Aware Gated Attention Network for Audio-Visual Event Localization
- 3 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 1601-1612
Abstract
Audio-visual event localization (AVEL) refers to the identification of the category and the corresponding temporal boundaries of an event that is both visually and audibly discernible in unconstrained videos. However, the event-irrelevant background (e.g., ambient noise or visual occlusion) and event-specific modal biases often lead to audio-visual semantic inconsistency. Existing methods utilize modality-guided attention to suppress background interference, but they neglect this attention inevitably introduces redundant or irrelevant information from the other modality. To alleviate this problem, we propose a novel Modality-Aware Gated Attention Network (MAGAN) that focuses on event-relevant visual regions, consolidates informative audio frequencies, and captures event-specific modality biases. Specifically, a cross-modal gated co-attention (CMGCA) scheme is presented for modeling the correspondence between the potential (self-guided) localization maps and the modality-guided localization maps through two gated components, i.e., audio-to-visual attention and visual-to-audio attention. Furthermore, a cross-modal gated co-interaction (CMGCI) mechanism that incorporates both unimodal gated interaction and multimodal gated interaction is introduced to capture event-specific modality biases by considering unimodal independence and multimodal synergy simultaneously. Extensive experiments on the AVE dataset demonstrate the superiority and effectiveness of our model over state-of-the-art approaches in both fully- and weakly-supervised AVE settings.Keywords
Funding Information
- Science and Technology Innovation Key R&D Program of Chongqing (CSTB2025TIAD-STX0030)
- Chongqing Natural Science Foundation Innovation and Development Joint Foundation (CSTB2024NSCQ-LZX0035)
- Science and Technology Research Project of Chongqing Education Commission (KJZD-M202300605)
- New Chongqing Youth Innovation Talent Plan Project (CSTB2024NSCQ-QCXMX0053)
- Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0073)
- Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0101)
- Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0027)
- Yongjiang Program for Young Talents (RC20230107)
- Chongqing Research Institution Performance Incentive Guidance Special Project (CSTB2023JXJL-YFX0013)
- Chongqing University of Posts and Telecommunications Ph.D. Innovative Talent Project (BYJS202409)
This publication has 58 references indexed in Scilit:
- Audio–Visual Segmentation2022
- Multimodal Attentive Fusion Network for audio-visual event recognitionInformation Fusion, 2022
- Deep Audio-visual Learning: A SurveyInternational Journal of Automation and Computing, 2021
- Audiovisual Transformer with Instance Attention for Audio-Visual Event Localization2021
- Do We Need Sound for Sound Source Localization?2021
- Cross-modal Deep Learning Applications: Audio-Visual Retrieval2021
- Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing2020
- Audio-Visual Event Localization in Unconstrained Videos2018
- The Sound of Pixels2018
- W-TALC: Weakly-Supervised Temporal Activity Localization and Classification2018