RadioSES: mmWave-Based Audioradio Speech Enhancement and Separation System
- 1 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 1333-1347
Abstract
Speech enhancement and separation have been a long-standing problem, especially with the recent advances using a single microphone. Although microphones perform well in constrained settings, their performance for speech separation decreases in noisy conditions. In this work, we propose RadioSES , an audioradio speech enhancement and separation system that overcomes inherent problems in audio-only systems. By fusing a complementary radio modality, RadioSES can estimate the number of speakers, solve the source association problem, separate and enhance noisy mixture speeches, and improve both intelligibility and perceptual quality. We perform millimeter-wave sensing to detect and localize speakers and introduce an audioradio deep learning framework to fuse the separate radio features with the mixed audio features. Extensive experiments using commercial off-the-shelf devices show that RadioSES outperforms a variety of state-of-the-art baselines, with consistent performance gains in different environmental settings. Similar to the audiovisual methods, RadioSES provides significant performance improvements 3 dB gains in SiSDR, when compared with the corresponding audio-only method, along with the benefits of lower computational complexity and better privacy preservation.Keywords
All Related Versions
Funding Information
- Key Bridge Foundation
This publication has 73 references indexed in Scilit:
- A Pairwise Algorithm Using the Deep Stacking Network for Speech Separation and Pitch EstimationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016
- Deep clustering: Discriminative embeddings for segmentation and separation2016
- Complex Ratio Masking for Monaural Speech SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- Smart Homes that Monitor Breathing and Heart Rate2015
- On Training Targets for Supervised Speech SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014
- Visual Input Enhances Selective Speech Envelope Tracking in Auditory Cortex at a “Cocktail Party”The Journal of Neuroscience, 2013
- Visual voice activity detection as a help for speech source separation from convolutive mixturesSpeech Communication, 2007
- Computational auditory scene analysisComputer Speech & Language, 1994
- Algorithms for the Assignment and Transportation ProblemsJournal of the Society for Industrial and Applied Mathematics, 1957
- Some Experiments on the Recognition of Speech, with One and with Two EarsThe Journal of the Acoustical Society of America, 1953