Sequence-to-Sequence Neural Diarization With Automatic Speaker Detection and Representation
- 19 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2719-2734
Abstract
This paper proposes a novel Sequence-to-Sequence Neural Diarization (S2SND) framework to perform online and offline speaker diarization. It is developed from the sequence-to-sequence architecture of our previous target-speaker voice activity detection system and then evolves into a new diarization paradigm by addressing two critical problems. 1) Speaker Detection: The proposed approach can utilize partially given speaker embeddings to discover the unknown speaker and predict the target voice activities in the audio signal. It does not require a prior diarization system for speaker enrollment in advance. 2) Speaker Representation: The proposed approach can adopt the predicted voice activities as reference information to extract speaker embeddings from the audio signal simultaneously. The representation space of speaker embedding is jointly learned within the whole diarization network without using an extra speaker embedding model. During inference, the S2SND framework can process long audio recordings blockwise. The detection module utilizes the previously obtained speaker-embedding buffer to predict both enrolled and unknown speakers' voice activities for each coming audio block. Next, the speaker-embedding buffer is updated according to the predictions of the representation module. Assuming that up to one new speaker may appear in a small block shift, our model iteratively predicts the results of each block and extracts target embeddings for the subsequent blocks until the signal ends. Finally, the last speaker-embedding buffer can re-score the entire audio, achieving highly accurate diarization performance as an offline system. Experimental results show that our proposed S2SND framework achieves new state-of-the-art diarization error rates (DERs) for online inference on the DIHARD-II (24.41%) and DIHARD-III (17.12%) evaluation sets without using oracle voice activity detection. At the same time, it also refreshes the state-of-the-art performance for offline inference on these benchmarks, with DERs of 21.95% and 15.13%, respectively.Keywords
Funding Information
- National Natural Science Foundation of China (62171207)
- Yangtze River Delta Science and Technology Innovation Community Joint Research (2024CSJGG01100)
- Science and Technology Program of Suzhou City (SYC2022051)
- Guangdong Science and Technology Plan (2023A1111120012)
This publication has 82 references indexed in Scilit:
- The DKU-MSXF Diarization System for the VoxCeleb Speaker Recognition Challenge 20232024
- A review of speaker diarization: Recent advances with deep learningComputer Speech & Language, 2022
- Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasksComputer Speech & Language, 2022
- Overlap-Aware Low-Latency Online Speaker Diarization Based on End-to-End Local Segmentation2021
- Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors2021
- End-to-End Neural Speaker Diarization with Self-Attention2019
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019
- A study on data augmentation of reverberant speech for robust speech recognition2017
- Convolutional Neural Network for speaker change detection in telephone speaker diarization system2017
- Deep Residual Learning for Image Recognition2016