Cross-Modal Feature Enhancement Networks for Audio-Visual Event Localization
- 4 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 25 (8) , 14393-14404
Abstract
The task of Audio-Visual Event (AVE) localization involves the temporal localization of both audible and visible events captured by camera sensors. However, the audio noise and visual background inherent in these sensors can obscure the distinction between event categories, or the inconsistency between audio and visual information may lead to incorrect predictions of event boundaries. To address these challenges, we propose a cross-modal feature enhancement network designed to enhance modality-specific audio and visual features as well as overall audio-visual event features. Specifically, our modality-specific feature enhancement approach employs cross-modal attentional mechanisms that allow the visual and audio modalities to focus on event-related features within the opposite modality. This approach also establishes intra-modal dependencies within each modality, enhancing the temporal relevance of event-related audio and visual features. Additionally, our audio-visual event feature enhancement scheme develops inter-modal correspondences through audio-visual interaction fusion, aligning the feature representations of audio-visual events temporally. To further mitigate the interference of visual background noise, we have developed an audio-visual channel-spatial fusion module that leverages audio-guided visual attention to improve the visual representation of the sounding target within the visual scene. Experiments conducted on the AVE dataset demonstrate that our method surpasses state-of-the-art methods in terms of AVE localization accuracy in both supervised and weakly supervised settings.Keywords
Funding Information
- National Natural Science Foundation of China (52374165)
- Student Innovation Training Program of China University of Mining and Technology-Beijing (202414016, 202414020)
This publication has 34 references indexed in Scilit:
- Dual Perspective Network for Audio-Visual Event Localization2022
- Positive Sample Propagation along the Audio-Visual Event Line2021
- Audiovisual Transformer with Instance Attention for Audio-Visual Event Localization2021
- Multiple Sound Sources Localization from Coarse to Fine2020
- Audio-Visual Event Localization in Unconstrained Videos2018
- Squeeze-and-Excitation Networks2018
- Learning to Localize Sound Source in Visual Scenes2018
- Non-local Neural Networks2018
- Deep Residual Learning for Image Recognition2016
- Explicit Encoding of Multimodal Percepts by Single Neurons in the Human BrainCurrent Biology, 2009