End-to-End Multi-Modal Speaker Change Detection with Pre-Trained Models
Open Access
- 14 April 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (8) , 4324
Abstract
In this work, we propose a multi-modal speaker change detection (SCD) approach with focal loss, which integrates both audio and text features to enhance detection performance. The proposed approach utilizes pre-trained large-scale models for feature extraction and incorporates a self-attention mechanism to optimize useful features related to speaker change. The extracted features are fused and processed through a fully connected classification network, with layer normalization and dropout for stability and generalization. To address class imbalance, we apply focal loss, which reduces errors for the difficult samples, leading to better balanced performance. Extensive experiments on a multi-talker meeting dataset demonstrate that the proposed multi-modal approach consistently outperforms single-modal models, proving the complementary nature of audio and text for SCD. Fine-tuning pre-trained models (Wav2Vec2 and Bert) for audio and text significantly boosts accuracy, achieving a 21% improvement over frozen models. The self-attention mechanism further improves performance by 2%, highlighting its ability to capture speaker transition cues effectively. Additionally, focal loss enhances the model’s performance, making it more robust to imbalanced data.Keywords
All Related Versions
Funding Information
- Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan (AP19676744)
This publication has 28 references indexed in Scilit:
- Speaker Change Detection Using Fundamental Frequency with Application to Multi-talker Segmentation2019
- Pre-training of Speaker Embeddings for Low-latency Speaker Change Detection in Broadcast News2019
- Architectural Approaches for Phonemes Recognition Systems2018
- TristouNet: Triplet loss for speaker turn embedding2017
- Speaker segmentation using deep speaker vectors for fast speaker change scenarios2017
- Variational conditional random fields for online speaker detection and trackingSpeech Communication, 2012
- Front-End, Back-End, and Hybrid Techniques for Noise-Robust Speech Recognition2011
- Kullback-Leibler Divergence2011
- The AMI Meeting Corpus: A Pre-announcement2006
- Segregation of speakers for speech recognition and speaker identification1991