End-to-End Multi-Modal Speaker Change Detection with Pre-trained Models
Preprint
- 12 March 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
In this work, we propose a multi-modal Speaker Change Detection (SCD) approach with focal loss that integrates both audio and text features to enhance detection performance. % The proposed approach utilizes pre-trained large-scale models for feature extraction and incorporates a self-attention mechanism to optimize the useful features related to speaker change. % The extracted features are fused and processed through a fully connected classification network, with layer normalization and dropout for stability and generalization. % To address class imbalance, we apply focal loss, which reduces errors for the difficult samples, leading to better balanced performance. Extensive experiments on a multi-talker meeting dataset demonstrate that the proposed multi-modal approach consistently outperforms single-modal models, proving the complementary nature of audio and text for SCD. % Fine-tuning pre-trained models (Wav2Vec2 and Bert) for audio and text significantly boosts accuracy, achieving a 21\% improvement over frozen models. % The self-attention mechanism further improves performance by 2\%, highlighting its ability to capture speaker transition cues effectively. % Additionally, focal loss enhances model's performance, making it more robust to imbalanced data.Keywords
All Related Versions
- Version 1, 2025-03-12, Preprints
- Published version: Applied Sciences, 15 (8), 4324.
This publication has 17 references indexed in Scilit:
- Comparative Analysis of Distance Measures for Unsupervised Speaker Change Detection2024
- Augmenting Transformer-Transducer Based Speaker Change Detection with Token-Level Training Loss2023
- Hybrid Speech and Text Analysis Methods for Speaker Change DetectionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Speaker Change Detection Using Fundamental Frequency with Application to Multi-talker Segmentation2019
- Pre-training of Speaker Embeddings for Low-latency Speaker Change Detection in Broadcast News2019
- Speaker Recognition from Raw Waveform with SincNet2018
- TristouNet: Triplet loss for speaker turn embedding2017
- Speaker segmentation using deep speaker vectors for fast speaker change scenarios2017
- Kullback-Leibler Divergence2011
- Segregation of speakers for speech recognition and speaker identification1991