Neural decoding of speech using deep neural ensembles
Authors:
- Seonghyun YoonSYSeonghyun Yoon
- Donald T. AvansinoDADonald T. Avansino
- Sasidhar MadugulaSMSasidhar Madugula
- Alisa D. LevinALAlisa D. Levin
- Chaofei FanCFChaofei Fan
- Benyamin Abramovich KrasaBKBenyamin Abramovich Krasa
- Akansha SinghASAkansha Singh
- Nick V. HahnNHNick V. Hahn
- Nicholas S. CardNCNicholas S. Card
- Maitreyee WairagkarMWMaitreyee Wairagkar
Preprint
Abstract
Speech brain-computer interfaces (BCIs) can restore rapid communication to people with paralysis, but decoding errors still limit performance. In recent brain-to-text decoding competitions, deep ensemble methods, which combine predictions from multiple independently trained decoders, have delivered striking accuracy improvements and account for the largest gains over baseline approaches. However, these methods have not previously been tested in real-time, require substantial computational resources, and their performance under various clinically relevant constraints remains poorly understood. Here, we present the first closed-loop test of deep ensembles in a participant with bilateral intracortical microelectrode arrays, demonstrating a reduction in word error rate from 33.7% to 26.0% on a large-vocabulary task. Using additional data from three participants, we then assess how these gains depend on baseline error rate, training dataset size, and ensemble size, including the resource-accuracy tradeoffs most relevant for real-world deployment. Finally, we introduce a computationally efficient pseudoensembling approach based on test-time augmentation that improves decoding accuracy while requiring only a single base decoder, greatly reducing the computational burden of ensembling. Together, these results show that the benefits of deep ensembling can be realized in real time and under practical resource constraints, bringing speech BCIs closer to broader clinical adoption.Keywords
Funding Information
- Office of the Assistant Secretary of Defense for Health Affairs (AL220043)
- National Institutes of Health (NIH 1DP2DC021055)
- National Institutes of Health (NIH F32HD112173)
- National Institutes of Health (NIH-NINDS/OD DP2NS127291)
- Simons Collaboration for the Global Brain (872146SPI)
- U.S. Department of Veterans Affairs (N2864C)
- U.S. Department of Veterans Affairs (A2295R)
- U.S. Department of Veterans Affairs (A4820R)
- Wu Tsai Neurosciences Institute, Stanford University
- Howard Hughes Medical Institute
- Larry and Pamela Garlick
- National Institute on Deafness and Other Communication Disorders (U01DC017844)
- National Institute on Deafness and Other Communication Disorders (R01DC014034)
- Burroughs Wellcome Fund
- Simons Foundation
- National Science Foundation
- A.P. Giannini Foundation
- Bio-X Interdisciplinary Biosciences Institute, Stanford University
- Searle Scholars Program
This publication has 68 references indexed in Scilit:
- Decoding intended speech with an intracortical brain-computer interface in a person with long-standing anarthria and locked-in syndromeCell Reports, 2026
- Moving beyond the motor cortex: A brain-wide evaluation of target locations for intracranial speech neuroprosthesesCell Reports, 2025
- Inner speech in motor cortex and implications for speech neuroprosthesesCell, 2025
- Imagined speech can be decoded from low- and cross-frequency intracranial EEG featuresNature Communications, 2022
- Real-time decoding of question-and-answer speech dialogue using human cortical activityNature Communications, 2019
- Accurate Estimation of Neural Population Dynamics without Spike SortingNeuron, 2019
- Automatic Brain Tumor Segmentation Using Convolutional Neural Networks with Test-Time Augmentation2019
- A multi-modal parcellation of human cerebral cortexNature, 2016
- Non-causal spike filtering improves decoding of movement intention for intracortical BCIsJournal of Neuroscience Methods, 2014
- Ensemble Methods in Machine Learning2000