Brain2voice 2.0: High-performance voice synthesis brain-computer interface
Authors:
- Maitreyee WairagkarMWMaitreyee Wairagkar
- Aparna SrinivasanASAparna Srinivasan
- Nicholas S CardNCNicholas S Card
- Tyler Singer-ClarkTSTyler Singer-Clark
- Xianda HouXHXianda Hou
- Carrina IacobacciCICarrina Iacobacci
- Lee M MillerLMLee M Miller
- Leigh R HochbergLHLeigh R Hochberg
- David M BrandmanDBDavid M Brandman
- Sergey D StaviskySSSergey D Stavisky
Preprint
Abstract
Brain-computer interfaces (BCIs) offer a promising solution to speech loss due to neurological injury by decoding intended speech directly from brain activity. While recent BCIs have restored high-accuracy text-based communication, they fail to provide instantaneous voice output essential for the natural flow of conversation. Brain-to-voice BCIs address this gap by decoding voice directly from neural signals. However, even the state-of-the-art (SOTA) BCI-synthesized voice is not yet intelligible enough for real-world adoption. We introduce brain2voice 2.0, a new multimodal Transformer-based BCI decoder architecture capable of synthesizing highly intelligible voice from intracortical neural signals in real-time. Brain2voice 2.0 is trained on continuous and custom-tokenized acoustic targets and phoneme targets, leveraging their complementary speech information. We use self-supervised and adversarial training objectives that enhance acoustic feature quality and improve synthesis intelligibility. At each 10 ms timestep, the model causally outputs continuous and tokenized acoustic features for real-time voice synthesis as well as time-aligned phoneme predictions (raw phoneme error rate: 7%, comparable to the latest brain-to-text models). We evaluated this new approach on our prior intracortical brain-to-voice benchmark dataset (Wairagkar et al. 2025). Naive human listeners transcribed brain2voice 2.0 synthesized voice with a word error rate of 5.24%—an 8× improvement in intelligibility over previous SOTA results (43.75%). Brain2voice 2.0 demonstrates that highly intelligible real-time voice synthesis from neural signals is achievable, for the first time crossing the intelligibility threshold necessary for clinically viable brain-to-voice BCIs for people with paralysis.Keywords
Funding Information
- Department of Defense CDMRP ALS Pilot Clinical Trial Award (AL220043)
- Department of Defense CDMRP ALS Pilot Clinical Trial Award (AL250417)
- Searle Scholars Program
- Career Award at the Scientific Interface from the Burroughs Wellcome Fund to Stavisky
- A.P. Giannini Postdoctoral Fellowship to Card
- Career Award at the Scientific Interface from the Burroughs Wellcome Fund to Card
- UC Davis School of Medicine Cultivating Team Science Award
- NIH/NCATS: UC Davis Clinical and Translational Science Center (CTSC UL1-TR001860)
- NIH-NIDCD (U01DC017844)
- VA RR&D (A2295-R)
This publication has 33 references indexed in Scilit:
- Decoding intended speech with an intracortical brain-computer interface in a person with long-standing anarthria and locked-in syndromeCell Reports, 2026
- Inner speech in motor cortex and implications for speech neuroprosthesesCell, 2025
- An instantaneous voice-synthesis neuroprosthesisNature, 2025
- Human motor cortex encodes complex handwriting through a sequence of stable neural statesNature Human Behaviour, 2025
- A streaming brain-to-voice neuroprosthesis to restore naturalistic communicationNature Neuroscience, 2025
- Online speech synthesis using a chronically implanted brain–computer interface in an individual with ALSScientific Reports, 2024
- A high-performance speech neuroprosthesisNature, 2023
- A high-performance neuroprosthesis for speech decoding and avatar controlNature, 2023
- High-performance brain-to-text communication via handwritingNature, 2021
- Speech synthesis from neural decoding of spoken sentencesNature, 2019