mmMIC: Multi-modal Speech Recognition based on mmWave Radar
Authors:
- 17 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
With the proliferation of voice assistants, microphone-based speech recognition technology usually cannot achieve good performance in the situation of multiple sound sources and ambient noises. In this paper, we propose a novel mmWave-based solution to perform speech recognition to tackle the issues of multiple sound sources and ambient noises, by precisely extracting the multi-modal features from lip motion and vocal-cords vibration from the single channel of mmWave. We propose a difference-based method for feature extraction of lip motion to suppress the dynamic interference from body motion and head motion. We propose a speech detection method based on cross-validation of lip motion and vocal-cords vibration so as to avoid wasting computing resources on nonspeaking activities. We propose a multi-modal fusion framework for speech recognition by fusing the signal features from lip motion and vocal-cords vibration with the attention mechanism. We implemented a prototype system and evaluated the performance in real test-beds. Experiment results show that the average speech recognition accuracy is 92.8% in realistic environments.Keywords
Funding Information
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Nanjing University
This publication has 34 references indexed in Scilit:
- The Speechtransformer for Large-scale Mandarin Chinese Speech Recognition2019
- Precise Power Delay Profiling with Commodity Wi-FiIEEE Transactions on Mobile Computing, 2018
- RED: RFID-based Eccentricity Detection for High-speed Rotating Machinery2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Comparing Social Robot, Screen and Voice Interfaces for Smart-Home Control2017
- We can hear you with Wi-Fi!2014
- The visual microphoneACM Transactions on Graphics, 2014
- Frequency Domain Processing2008
- The Human VoiceScientific American, 1992
- Signal estimation from modified short-time Fourier transformIEEE Transactions on Acoustics, Speech, and Signal Processing, 1984