Dual-Strategy-Enhanced ConBiMamba for Neural Speaker Diarization
Preprint
Abstract
Conformer and Mamba have achieved strong performance in speech modeling but face limitations in speaker diarization. Mamba is efficient but struggles with local details and nonlinear patterns. Conformer's self-attention incurs high memory overhead for long speech sequences and may cause instability in long-range dependency modeling. These limitations are critical for diarization, which requires both precise modeling of local variations and robust speaker consistency over extended spans. To address these challenges, we first apply ConBiMamba for speaker diarization. We follow the Pyannote pipeline and propose the Dual-Strategy-Enhanced ConBiMamba neural speaker diarization system. ConBiMamba integrates the strengths of Conformer and Mamba, where Conformer's convolutional and feed-forward structures are utilized to improve local feature extraction. By replacing Conformer's self-attention with ExtBiMamba, ConBiMamba efficiently handles long audio sequences while alleviating the high memory cost of self-attention. Furthermore, to address the problem of the higher DER around speaker change points, we introduce the Boundary-Enhanced Transition Loss to enhance the detection of speaker change points. We also propose Layer-wise Feature Aggregation to enhance the utilization of multi-layer representations. The system is evaluated on six diarization datasets and achieves state-of-the-art performance on four of them. The source code of our study is available at https://github.com/lz-hust/DSE-CBM.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Mamba in Speech: Towards an Alternative to Self-AttentionIEEE Transactions on Audio, Speech and Language Processing, 2025
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- M2Met: The Icassp 2022 Multi-Channel Multi-Party Meeting Transcription Challenge2022
- AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario2021
- Advances in Integration of End-to-End Neural and Clustering-Based Diarization for Real Conversational Speech2021
- ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification2020
- End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors2020
- Hierarchical Grouping to Optimize an Objective FunctionJournal of the American Statistical Association, 1963