Simultaneous speech and gesture decoding for multimodal communication in paralysis
Open Access
- 14 September 2026
- journal article
- research article
- Published by Springer Nature in Nature Neuroscience
- Vol. 29 (10) , 2445-2455
Abstract
Stroke and neurodegenerative diseases can impair speech and nonverbal gestures, limiting natural communication. Brain–computer interfaces (BCIs) aim to restore these functions by translating neural activity into commands for external devices, although prior work has primarily focused on decoding speech or gestures in isolation. Here we show that neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech and gestures in people with paralysis. We first show that isolated upper-limb and orofacial movements can be reliably decoded among three participants. Using parallel speech and gesture decoders, we then enabled participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. Training models on both isolated and simultaneous data improved performance across behavioral contexts. These findings demonstrate that one cortical implant can support multi-effector control and provide a step toward BCIs that enable more natural communication for people with paralysis.Keywords
Funding Information
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- Japan Science and Technology Agency (JPMJMS2012)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
- National Institute on Deafness and Other Communication Disorders (5U01DC018671)
This publication has 59 references indexed in Scilit:
- Optimal location for gesture decoding in the sensorimotor cortex and implications for brain-computer interface researchNeuroImage, 2026
- Brain–computer interfaces in 2023–2024Brain‐X, 2025
- Sampling representational plasticity of simple imagined movements across days enables long-term neuroprosthetic controlCell, 2025
- Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 MonthsAdvanced Science, 2023
- Hand Knob Area of Premotor Cortex Represents the Whole Body in a Compositional WayCell, 2020
- Sub-millimeter fMRI reveals multiple topographical digit representations that form action maps in human motor cortexNeuroImage, 2020
- Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor CortexNeuron, 2018
- Give me a sign: decoding four complex hand gestures based on high-density ECoGBrain Structure and Function, 2014
- Interactions between areas of the cortical grasping networkCurrent Opinion in Neurobiology, 2011
- Mapping Behavioral Repertoire onto the CortexNeuron, 2007