A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages
Top Cited Papers
- 20 May 2024
- journal article
- research article
- Published by Springer Nature in Nature Biomedical Engineering
- Vol. 8 (8) , 977-991
Abstract
Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish–English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.Keywords
Funding Information
- Foundation for the National Institutes of Health (DC018671-01A1, DC018671-01A1, DC018671-01A1, DC018671-01A1, DC018671-01A1, DC018671-01A1)
- U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (T32GM007618)
This publication has 71 references indexed in Scilit:
- Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 MonthsAdvanced Science, 2023
- Stability of a chronic implanted brain-computer interface in late-stage amyotrophic lateral sclerosisClinical Neurophysiology, 2019
- A Survey on Deep Transfer Learning2018
- A critical period for second language acquisition: Evidence from 2/3 million English speakersCognition, 2018
- Anarthria2017
- New Era for Robust Speech Recognition2017
- Brain-Computer Interface Users Speak Up: The Virtual Users' Forum at the 2013 International Brain-Computer Interface MeetingArchives of Physical Medicine and Rehabilitation, 2015
- Speaking words in two languages with one brain: Neural overlap and dissociationBrain Research, 2010
- Review of language organisation in bilingual patients: what can we learn from direct brain mapping?Acta Neurochirurgica, 2007
- The neural basis of first and second language processingCurrent Opinion in Neurobiology, 2005