Abstract
The task of Audio-Visual Event (AVE) localization involves the temporal localization of both audible and visible events captured by camera sensors. However, the audio noise and visual background inherent in these sensors can obscure the distinction between event categories, or the inconsistency between audio and visual information may lead to incorrect predictions of event boundaries. To address these challenges, we propose a cross-modal feature enhancement network designed to enhance modality-specific audio and visual features as well as overall audio-visual event features. Specifically, our modality-specific feature enhancement approach employs cross-modal attentional mechanisms that allow the visual and audio modalities to focus on event-related features within the opposite modality. This approach also establishes intra-modal dependencies within each modality, enhancing the temporal relevance of event-related audio and visual features. Additionally, our audio-visual event feature enhancement scheme develops inter-modal correspondences through audio-visual interaction fusion, aligning the feature representations of audio-visual events temporally. To further mitigate the interference of visual background noise, we have developed an audio-visual channel-spatial fusion module that leverages audio-guided visual attention to improve the visual representation of the sounding target within the visual scene. Experiments conducted on the AVE dataset demonstrate that our method surpasses state-of-the-art methods in terms of AVE localization accuracy in both supervised and weakly supervised settings.
Funding Information
  • National Natural Science Foundation of China (52374165)
  • Student Innovation Training Program of China University of Mining and Technology-Beijing (202414016, 202414020)