EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech Enhancement
Open Access
- 22 August 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 8 (3) , 1-30
Abstract
Earphones have become a popular voice input and interaction device. However, airborne speech is susceptible to ambient noise, making it necessary to improve the quality and intelligibility of speech on earphones in noisy conditions. As the dual-microphone structure (i.e., outer and in-ear microphones) has been widely adopted in earphones (especially ANC earphones), we design EarSpeech which exploits in-ear acoustic sensory as the complementary modality to enable airborne speech enhancement. The key idea of EarSpeech is that in-ear speech is less sensitive to ambient noise and exhibits a correlation with airborne speech. However, due to the occlusion effect, in-ear speech has limited bandwidth, making it challenging to directly correlate with full-band airborne speech. Therefore, we exploit the occlusion effect to carry out theoretical modeling and quantitative analysis of this cross-channel correlation and study how to leverage such cross-channel correlation for speech enhancement. Specifically, we design a series of methodologies including data augmentation, deep learning-based fusion, and noise mixture scheme, to improve the generalization, effectiveness, and robustness of EarSpeech, respectively. Lastly, we conduct real-world experiments to evaluate the performance of our system. Specifically, EarSpeech achieves an average improvement ratio of 27.23% and 13.92% in terms of PESQ and STOI, respectively, and significantly improves SI-SDR by 8.91 dB. Benefiting from data augmentation, EarSpeech can achieve comparable performance with a small-scale dataset that is 40 times less than the original dataset. In addition, we validate the generalization of different users, speech content, and language types, respectively, as well as robustness in the real world via comprehensive experiments. The audio demo of EarSpeech is available on https://github.com/EarSpeech/earspeech.github.io/.Keywords
Funding Information
- National Natural Science Foundation of China (No. 62072004)
This publication has 41 references indexed in Scilit:
- Accuth+: Accelerometer-Based Anti-Spoofing Voice Authentication on Wrist-Worn WearablesIEEE Transactions on Mobile Computing, 2023
- RadioSES: mmWave-Based Audioradio Speech Enhancement and Separation SystemIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- STFT-Domain Neural Speech Enhancement With Very Low Algorithmic LatencyIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- On the removal of the open earcanal high-pass filter effect due to its occlusion: A bone-conduction occlusion effect theoryActa Acustica, 2021
- Relationship between phoneme-level spectral acoustics and speech intelligibility in healthy speech: a systematic reviewSpeech, Language and Hearing, 2021
- An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and SeparationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- A Novel Encoder-Decoder Model via NS-LSTM Used for Bone-Conducted Speech EnhancementIEEE Access, 2018
- Speech enhancement with an adaptive Wiener filterInternational Journal of Speech Technology, 2013
- Voice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter TrajectoryIEEE Transactions on Audio, Speech, and Language Processing, 2007
- A signal subspace approach for speech enhancementIEEE Transactions on Speech and Audio Processing, 1995