Audioradio Target Speech Detection and Extraction with mmWave Sensing
Open Access
- 4 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1796-1798
Abstract
Detecting the target speaker and enhancing speech with high fidelity has been a long-standing problem, especially in challenging acoustic conditions. To address these problems with minimal user cooperation, we have developed multimodal audioradio speech detection (RadioVAD) and enhancement (RadioSES) systems using mmWave modality. This demo presents how these two systems can be run together in real time to extract target speech and filter any type of ambient noise. Our demo uses an mmWave radar and a microphone attached to a laptop to detect and localize target speakers in the field of view, detect the presence of voice to trigger the microphone, and enhance the noisy speech with a deep learning model running in real-time. Our experiments confirm that an audioradio system can detect and isolate high-fidelity target speech, even with interfering speech and noise; while being privacy preserving and environmentally robust.Keywords
This publication has 7 references indexed in Scilit:
- RadioVAD: mmWave-Based Noise and Interference-Resilient Voice Activity DetectionIEEE Internet of Things Journal, 2024
- RadioSES: mmWave-Based Audioradio Speech Enhancement and Separation SystemIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- RadioMic: Sound Sensing via Radio SignalsIEEE Internet of Things Journal, 2022
- Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition2022
- An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation2020
- Looking to listen at the cocktail partyACM Transactions on Graphics, 2018