RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing
Authors:
- Efthymios TzinisETEfthymios Tzinis
- Yossi AdiYAYossi Adi
- Vamsi Krishna IthapuVIVamsi Krishna Ithapu
- Buye XuBXBuye Xu
- Paris SmaragdisPSParis Smaragdis
- Anurag KumarAKAnurag Kumar
Preprint
Abstract
We present RemixIT, a simple yet effective self-supervised method for training speech enhancement without the need of a single isolated in-domain speech nor a noise waveform. Our approach overcomes limitations of previous methods which make them dependent on clean in-domain target signals and thus, sensitive to any domain mismatch between train and test samples. RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures. Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models. Experimental results on multiple speech enhancement datasets and tasks not only show the superiority of our method over prior approaches but also showcase that RemixIT can be combined with any separation model as well as be applied towards any semi-supervised and unsupervised domain adaptation task. Our analysis, paired with empirical evidence, sheds light on the inside functioning of our self-training scheme wherein the student model keeps obtaining better performance while observing severely degraded pseudo-targets.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Incorporating Real-World Noisy Speech in Neural-Network-Based Speech Enhancement Systems2021
- Compute and Memory Efficient Universal Sound Source SeparationJournal of Signal Processing Systems, 2021
- Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement2021
- Robust Speaker Recognition Based on Single-Channel and Multi-Channel Speech EnhancementIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Unsupervised Speech Enhancement Based on Multichannel NMF-Informed Beamforming for Noise-Robust Automatic Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019
- Two-Stage Deep Learning for Noisy-Reverberant Speech EnhancementIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018
- Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization2018
- Permutation invariant training of deep models for speaker-independent multi-talker speech separationMechanical Engineering Journal, 2017
- Multichannel Signal Processing With Deep Neural Networks for Automatic Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2017
- Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs2002