Adversarial Contrastive Autoencoder With Shared Attention for Audio-Visual Correlation Learning
Open Access
- 28 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 39753-39764
Abstract
Cross-modal audio-visual correlation learning has been an active research topic, which aims to embed audio and visual feature sequences into a common subspace where their correlation is maximized. The challenge of audio-visual correlation learning lies in two major aspects: i) audio and visual feature sequences respectively contain different patterns belonging to different feature spaces, and ii) semantic mismatch between audio and visual sequences inevitably happens during cross-modal matching. Most existing methods only take the first aspect into account, therefore facing the difficulty in distinguishing matched and mismatched semantic correlations between audio and visual sequences. In this work, an adversarial contrastive autoencoder with a shared attention network (ACASA) is proposed for correlation learning in audio-visual retrieval. In particular, the proposed shared attention mechanism is parameterized, in which local salient information is enhanced to contribute to the final feature representation. Simultaneously, adversarial contrastive learning is exploited to maximize semantic feature consistency and improve the ability to distinguish matched and mismatched samples. Both inter-modal and intra-modal semantic information are utilized to supervise the model to learn more discriminative feature representation. Extensive experiments on the VEGAS and AVE datasets demonstrate that the proposed ACASA method outperforms state-of-the-art approaches in cross-modal audio-visual retrieval.Keywords
This publication has 39 references indexed in Scilit:
- Multi-scale network with shared cross-attention for audio–visual correlation learningNeural Computing and Applications, 2023
- Unsupervised Cross‐Media Hashing Learning via Knowledge GraphChinese Journal of Electronics, 2022
- Self-supervised Learning of Audio-Visual Objects from Video2020
- Modality-specific and shared generative adversarial network for cross-modal retrievalPattern Recognition, 2020
- Cross-modal Embeddings for Video and Audio Retrieval2019
- Audio-Visual Event Localization in Unconstrained Videos2018
- Stacked Cross Attention for Image-Text Matching2018
- Deep adversarial metric learning for cross-modal retrievalWorld Wide Web, 2018
- A Discriminative Feature Learning Approach for Deep Face Recognition2016
- Relations Between Two Sets of Variates1992