Modeling Speech Structure to Improve T-F Masks for Speech Enhancement and Recognition
- 3 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 30 (23299290) , 2705-2715
Abstract
Time-frequency (TF) masks are widely used in speech enhancement (SE). However, accurately estimating TF masks from noisy speech remains a challenge to both statistical or neural network (NN) approaches. Statistical model based mask estimation usually depends on a good parameter initialization, while NN-based method relies on setting proper and stable learning targets. To address these issues, we propose to extract TF speech structure from clean speech and partition noisy speech spectrogram into mutually exclusive regions. We investigate modeling clean speech by utterance-specific narrowband complex Gaussian mixture models to derive the regions, and using the region targets to supervise the training of UNet++, a high-performance NN, for predicting regions from noisy speech. For multichannel SE, we consider two scenarios of using speech regions: 1) integrating the regions with TF masks by constraining the mask values or the model parameter updates, and 2) using the predicted regions in place of TF masks. For single-channel SE, we consider using the region targets to improve TF mask targets. Furthermore, we propose to use UNet++ for TF mask estimation. Our experiment results on speech recognition (CHiME-3) and SE (CHiME-3 and LibriSpeech) have demonstrated the effectiveness of our proposed approach of modeling speech region structure to improve TF masks for speech recognition and enhancement.Keywords
Funding Information
- PAII
This publication has 50 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Librispeech: An ASR corpus based on public domain audio books2015
- Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks2015
- On Training Targets for Supervised Speech SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014
- On the Ideal Ratio Mask as the Goal of Computational Auditory Scene Analysis2014
- Ideal ratio mask estimation using deep neural networks for robust speech recognition2013
- Role of mask pattern in intelligibility of ideal binary-masked noisy speechThe Journal of the Acoustical Society of America, 2009
- Binary and ratio time-frequency masks for robust speech recognitionSpeech Communication, 2006
- Monaural Speech Segregation Based on Pitch Tracking and Amplitude ModulationIEEE Transactions on Neural Networks, 2004
- Auditory Scene Analysis: The Perceptual Organization of SoundThe Journal of the Acoustical Society of America, 1994