A streaming brain-to-voice neuroprosthesis to restore naturalistic communication
Top Cited Papers
- 31 March 2025
- journal article
- technical report
- Published by Springer Nature in Nature Neuroscience
- Vol. 28 (4) , 902-912
Abstract
Natural spoken communication happens instantaneously. Speech delays longer than a few seconds can disrupt the natural flow of conversation. This makes it difficult for individuals with paralysis to participate in meaningful dialogue, potentially leading to feelings of isolation and frustration. Here we used high-density surface recordings of the speech sensorimotor cortex in a clinical trial participant with severe paralysis and anarthria to drive a continuously streaming naturalistic speech synthesizer. We designed and used deep learning recurrent neural network transducer models to achieve online large-vocabulary intelligible fluent speech synthesis personalized to the participant’s preinjury voice with neural decoding in 80-ms increments. Offline, the models demonstrated implicit speech detection capabilities and could continuously decode speech indefinitely, enabling uninterrupted use of the decoder and further increasing speed. Our framework also successfully generalized to other silent-speech interfaces, including single-unit recordings and electromyography. Our findings introduce a speech-neuroprosthetic paradigm to restore naturalistic spoken communication to people with paralysis.Keywords
Funding Information
- Rose Hills and Noyce Foundation, the Joan and Sandy Weill Foundation, Susan and Bill Oberndorf, Ron Conway, Graham and Christina Spencer, and the William K. Bowes, Jr. Foundation.
- U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671, 5U01DC018671)
- MEXT | Japan Science and Technology Agency
- Rose Hills and Noyce Foundation, the Joan and Sandy Weill Foundation, Susan and Bill Oberndorf, Ron Conway, Graham and Christina Spencer, and the William K. Bowes, Jr. Foundation. NSF GFRP
- The Joan and Sandy Weill Foundation, Susan and Bill Oberndorf, Ron Conway, Graham and Christina Spencer, and the William K. Bowes, Jr. Foundation.
- U.S. Department of Health & Human Services | NIH | National Institute on Deafness and Other Communication Disorders (F30DC021872)
This publication has 54 references indexed in Scilit:
- Verbal communication impacts quality of life in patients with amyotrophic lateral sclerosisAmyotrophic Lateral Sclerosis and Frontotemporal Degeneration, 2015
- Brain-to-text: decoding spoken phrases from phone representations in the brainFrontiers in Neuroscience, 2015
- Brain-Computer Interface Users Speak Up: The Virtual Users' Forum at the 2013 International Brain-Computer Interface MeetingArchives of Physical Medicine and Rehabilitation, 2015
- Direct classification of all American English phonemes using signals from functional speech motor cortexJournal of Neural Engineering, 2014
- Functional organization of human sensorimotor cortex for speech articulationNature, 2013
- Human cortical sensorimotor network underlying feedback control of vocal pitchProceedings of the National Academy of Sciences, 2013
- What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosisAmyotrophic Lateral Sclerosis, 2011
- A Wireless Brain-Machine Interface for Real-Time Speech SynthesisPLOS ONE, 2009
- Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma bandBrain, 1998
- Articulatory model for the study of speech productionThe Journal of the Acoustical Society of America, 1973