AVCLNet: Multimodal Multispeaker Tracking Network Using Audio‐Visual Contrastive Learning
Open Access
- 29 November 2025
- journal article
- Published by Institution of Engineering and Technology (IET) in CAAI Transactions on Intelligence Technology
- Vol. 11 (1) , 238-255
Abstract
Audio‐visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms. Multimodal fusion methods can improve both the accuracy and robustness of speaker tracking. However, in complex multispeaker tracking scenarios, critical challenges such as cross‐modal feature discrepancy, weak sound source localisation ambiguity and frequent identity switch errors remain unresolved, which severely hinder the modelling of speaker identity consistency and consequently lead to degraded tracking accuracy and unstable tracking trajectories. To this end, this paper proposes a multimodal multispeaker tracking network using audio‐visual contrastive learning (AVCLNet). By integrating heterogeneous modal representations into a unified space through audio‐visual contrastive learning, which facilitates cross‐modal feature alignment, mitigates cross‐modal feature bias and enhances identity‐consistent representations. In the audio‐visual measurement stage, we design a vision‐guided weak sound source weighted enhancement method, which leverages visual cues to establish cross‐modal mappings and employs a spatiotemporal dynamic weighted mechanism to improve the detectability of weak sound sources. Furthermore, in the data association phase, a dual geometric constraint strategy is introduced by combining the 2D and 3D spatial geometric information, reducing frequent identity switch errors. Experiments on the AV16.3 and CAV3D datasets show that AVCLNet outperforms state‐of‐the‐art methods, demonstrating superior robustness in multispeaker scenarios.Keywords
Funding Information
- National Natural Science Foundation of China - State Grid Corporation Joint Fund for Smart Grid (62403345)
This publication has 39 references indexed in Scilit:
- Track Coalescence and Repulsion in Multitarget Tracking: An Analysis of MHT, JPDA, and Belief Propagation MethodsIEEE Open Journal of Signal Processing, 2024
- HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion RecognitionInformation Fusion, 2024
- PGSL: A probabilistic graph diffusion model for source localizationExpert Systems with Applications, 2023
- Acoustic Source Localization Using a Geometrically Sampled Grid SRP-PHAT Algorithm With Max-Pooling OperationIEEE Signal Processing Letters, 2022
- Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural NetworksIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics RecordingsIEEE Journal of Selected Topics in Signal Processing, 2019
- Audio-Visual Speech-Turn Detection and Tracking2015
- Speaker Diarization: A Review of Recent ResearchIEEE Transactions on Audio, Speech, and Language Processing, 2012
- Dynamical information fusion of heterogeneous sensors for 3D tracking using particle swarm optimizationInformation Fusion, 2010
- Target Detection and Tracking With Heterogeneous SensorsIEEE Journal of Selected Topics in Signal Processing, 2008