Contrastive Conditional Latent Diffusion for Audio-Visual Segmentation
Top Cited Papers
- 23 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 4108-4119
Abstract
Audio-visual Segmentation (AVS) is conceptualized as a conditional generation task, where audio is considered as the conditional variable for segmenting the sound producer(s). In this case, audio should be extensively explored to maximize its contribution for the final segmentation task. We propose a contrastive conditional latent diffusion model for audio-visual segmentation (AVS) to thoroughly investigate the impact of audio, where the correlation between audio and the final segmentation map is modeled to guarantee the strong correlation between them. To achieve semantic-correlated representation learning, our framework incorporates a latent diffusion model. The diffusion model learns the conditional generation process of the ground-truth segmentation map, resulting in ground-truth aware inference during the denoising process at the test stage. As our model is conditional, it is vital to ensure that the conditional variable contributes to the model output. We thus extensively model the contribution of the audio signal by minimizing the density ratio between the conditional probability of the multimodal data, e.g. conditioned on the audio-visual data, and that of the unimodal data, e.g. conditioned on the audio data only. In this way, our latent diffusion model via density ratio optimization explicitly maximizes the contribution of audio for AVS, which can then be achieved with contrastive learning as a constraint, where the diffusion part serves as the main objective to achieve maximum likelihood estimation, and the density ratio optimization part imposes the constraint. By adopting this latent diffusion model via contrastive learning, we effectively enhance the contribution of audio for AVS. The effectiveness of our solution is validated through experimental results on the benchmark dataset. Code and results are online via our project page: https://github.com/OpenNLPLab/DiffusionAVS.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62271410, 12150007)
- Fundamental Research Funds for the Central Universities
- Innovation Foundation for Doctoral Dissertation of Northwestern Polytechnical University (CX2024014)
This publication has 45 references indexed in Scilit:
- Instance-Level Moving Object Segmentation from a Single Image with EventsInternational Journal of Computer Vision, 2025
- Audio-Visual Segmentation with SemanticsInternational Journal of Computer Vision, 2024
- Audio–Visual Segmentation2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- Panoptic Feature Pyramid Networks2019
- Deep Residual Learning for Image Recognition2016
- FaceNet: A unified embedding for face recognition and clustering2015
- ImageNet: A large-scale hierarchical image database2009
- Dimensionality Reduction by Learning an Invariant Mapping2006
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005