Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation
Authors:
- Zhong-Qiu WangZWZhong-Qiu Wang
- Peidong WangPWPeidong Wang
- DeLiang WangDWDeLiang Wang
Preprint
Abstract
We propose multi-microphone complex spectral mapping, a simple way of applying deep learning for time-varying non-linear beamforming, for speaker separation in reverberant conditions. We aim at both speaker separation and dereverberation. Our study first investigates offline utterance-wise speaker separation and then extends to block-online continuous speech separation (CSS). Assuming a fixed array geometry between training and testing, we train deep neural networks (DNN) to predict the real and imaginary (RI) components of target speech at a reference microphone from the RI components of multiple microphones. We then integrate multi-microphone complex spectral mapping with minimum variance distortionless response (MVDR) beamforming and post-filtering to further improve separation, and combine it with frame-level speaker counting for block-online CSS. Although our system is trained on simulated room impulse responses (RIR) based on a fixed number of microphones arranged in a given geometry, it generalizes well to a real array with the same geometry. State-of-the-art separation performance is obtained on the simulated two-talker SMS-WSJ corpus and the real-recorded LibriCSS dataset.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- ADL-MVDR: All Deep Learning MVDR Beamformer for Target Speech Separation2021
- Continuous Speech Separation with Conformer2021
- Time–Frequency Masking Based Online Multi-Channel Speech Enhancement With Convolutional Recurrent Neural NetworksIEEE Journal of Selected Topics in Signal Processing, 2019
- Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained With Noise SignalsIEEE Journal of Selected Topics in Signal Processing, 2019
- Supervised Speech Separation Based on Deep Learning: An OverviewIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018
- The third ‘CHiME’ speech separation and recognition challenge: Analysis and outcomesComputer Speech & Language, 2017
- Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural NetworksIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- A Consolidated Perspective on Multimicrophone Speech Enhancement and Source SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- An Algorithm for Predicting the Intelligibility of Speech Masked by Modulated Noise MaskersIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016
- Complex Ratio Masking for Monaural Speech SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015