Enhancing Overlapped Speech Detection and Speaker Counting With Spatially-Infused Spectro-Temporal Conformer
- 24 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1307-1323
Abstract
Overlapped speech is widely present in conversations and can cause significant performance degradation in speech processing such as diarization, enhancement, and recognition. Overlapped speech detection (OSD) and speaker counting, in particular when the speakers are in the far field, pose challenges as the overlapped part is usually short, and heavy reverberation and noise may present in the conversation scenario. Existing solutions predominantly rely on spectral features extracted from a single microphone signal to perform the detection. In this paper, we propose an innovative approach that is able to efficiently extract spatial features via spatial dictionary learning (SDL) and fuse spatial and spectral features derived from multi-channel array signals. In essence, a spectro-temporal module is employed to extract local information from the spectrogram which is able to better detect overlapping segments under real scenarios. Subsequently, we study two kinds of spatial features: directional statistics, which are projected onto spherical location grids using SDL, and generalized cross-correlation with phase transform (GCC-PHAT) to model the spatial characteristics of speakers. These spatial features are then integrated with the spectral features for the OSD and speaker counting. In addition, a non-parameter magnitude-weighted fusion method is employed to combine the magnitude spectrogram with SDL results, enabling the model to prioritize critical frequency bins. Finally, the performance of the proposed approach is studied under AMI, Alimeeting, and CHiME-6 corpora. Experimental results show that our proposed approach achieves better performance than methods using spectral features only. Furthermore, we also conduct experiments of speaker diarization with OSD to evaluate the detection performance.Keywords
Funding Information
- National Natural Science Foundation of China (62376090)
- Hunan Provincial Natural Science Foundation of China (2022JJ20015)
This publication has 66 references indexed in Scilit:
- Audio-Visual Overlapped Speech Detection for Spontaneous Distant SpeechIEEE Access, 2023
- Overlapped Speech Detection and speaker counting using distant microphone arraysComputer Speech & Language, 2022
- Speaker Identification in Multi-Talker Overlapping Speech Using Neural NetworksIEEE Access, 2020
- End-to-End Overlapped Speech Detection and Speaker Counting with Raw Waveform2019
- Detection of Overlapping Speech for the Purposes of Speaker Diarization2019
- Blind speaker counting in highly reverberant environments by clustering coherence features2017
- Deep Learning-Based Feature Representation for AD/MCI Classification2013
- Efficient use of overlap information in speaker diarization2007
- The AMI Meeting Corpus: A Pre-announcement2006
- The Rich Transcription 2006 Spring Meeting Recognition Evaluation2006