Abstract
Audio-visual event localization (AVEL) refers to the identification of the category and the corresponding temporal boundaries of an event that is both visually and audibly discernible in unconstrained videos. However, the event-irrelevant background (e.g., ambient noise or visual occlusion) and event-specific modal biases often lead to audio-visual semantic inconsistency. Existing methods utilize modality-guided attention to suppress background interference, but they neglect this attention inevitably introduces redundant or irrelevant information from the other modality. To alleviate this problem, we propose a novel Modality-Aware Gated Attention Network (MAGAN) that focuses on event-relevant visual regions, consolidates informative audio frequencies, and captures event-specific modality biases. Specifically, a cross-modal gated co-attention (CMGCA) scheme is presented for modeling the correspondence between the potential (self-guided) localization maps and the modality-guided localization maps through two gated components, i.e., audio-to-visual attention and visual-to-audio attention. Furthermore, a cross-modal gated co-interaction (CMGCI) mechanism that incorporates both unimodal gated interaction and multimodal gated interaction is introduced to capture event-specific modality biases by considering unimodal independence and multimodal synergy simultaneously. Extensive experiments on the AVE dataset demonstrate the superiority and effectiveness of our model over state-of-the-art approaches in both fully- and weakly-supervised AVE settings.
Funding Information
  • Science and Technology Innovation Key R&D Program of Chongqing (CSTB2025TIAD-STX0030)
  • Chongqing Natural Science Foundation Innovation and Development Joint Foundation (CSTB2024NSCQ-LZX0035)
  • Science and Technology Research Project of Chongqing Education Commission (KJZD-M202300605)
  • New Chongqing Youth Innovation Talent Plan Project (CSTB2024NSCQ-QCXMX0053)
  • Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0073)
  • Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0101)
  • Special General Project for Chongqing’s Technological Innovation and Application Development (CSTB2024TIAD-KPX0027)
  • Yongjiang Program for Young Talents (RC20230107)
  • Chongqing Research Institution Performance Incentive Guidance Special Project (CSTB2023JXJL-YFX0013)
  • Chongqing University of Posts and Telecommunications Ph.D. Innovative Talent Project (BYJS202409)