NYNicholas Au Yong

Nicholas Au Yong

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Nicholas Au Yong has a total of 137 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Sentence-level neural dynamics in speech motor cortex

    by H. Levi-Aharoni, J. J. Jude, S. Haro, A. J. Acosta, S. B. Allcroft, M. C. Ottenhoff, N. S. Card, M. Wairagkar, C. Iacobacci, A. L. Pritchard, et al.

    Prior recordings from speech motor cortex have revealed phonemic, syllabic, articulatory and speech-sequence representations, while more recent studies indicate that activity may also reflect higher-order features of speech production. However, whether speech motor cortex is sensitive to the linguistic organization of sentences remains unclear. Here we show that single-unit activity in speech motor cortex contains information about sentence-level linguistic structure. In three BrainGate2 participants with implanted microelectrode arrays in the left precentral gyrus, we recorded neural activity during an instructed-delay speech task comprising English sentences, word lists, syntactically structured nonword sentences and nonword lists. Neural activity in areas 6v and 55b contained speech- and language-related components, with stronger stimulus-condition effects in 55b, and stimulus condition was decodable in all three participants. In two larger datasets, semantic categories of English sentences were decodable from neural activity. These findings demonstrate that speech motor cortex encodes information about linguistic structure and semantic content at the level of sentences.

  • Open Access

    A generalizable speech neuroprosthesis

    by Zachery M. Fogg, Nicholas S. Card, Maitreyee Wairagkar, Aparna Srinivasan, Tyler Singer-Clark, Xianda Hou, Elizaveta Okorokova, Hamza Peracha, Carrina Iacobacci, Tiffany Brailow, et al.

    Intracortical brain-computer interfaces (BCIs) can restore communication to people with vocal tract paralysis by decoding cortical activity during attempted speech into text. State-of-the-art systems pairing neural-to-phoneme decoders with phoneme-to-word language models have achieved word error rates (WERs) as low as 1%, but only after collecting thousands of sentences of training data. Shortening the data collection process would facilitate scaling this new technology by reducing the time from device implant to high-accuracy communication. Here we introduce a transformer-based decoder model trained jointly across six intracortical speech BCI participants. For every participant - regardless of sex, disease etiology, or attempted speaking strategy - a multi-user model decoded speech more accurately (over 50% lower relative WER on average) than models trained on individual users' data. Notably, the multi-user model could be fine-tuned on fewer than 200 sentences from a held-out user to achieve a WER below 7%. These results reveal how to pool intracortical data across people to yield more accurate, generalizable, and rapidly-deployable decoding models.


  • Top Cited Papers
    Open Access

    A mosaic of whole-body representations on the human precentral gyrus

    by Darrel R. Deo, Elizaveta V. Okorokova, Anna L. Pritchard, Nick V. Hahn, Nicholas S. Card, Samuel R. Nason-Tomaszewski, Justin Jude, Thomas Hosman, Eun Young Choi, Deqiang Qiu, et al.

    Understanding how the body is represented in the motor cortex is key to understanding how the brain controls movement. Although the motor cortex has been mapped in animal models at a fine scale1,2,3,4,5,6,7,8,9,10, characterization in humans remains primarily limited to low-resolution recording11,12,13,14,15,16 and stimulation techniques17,18,19,20. Here we created a comprehensive map of the human motor cortex at single-neuron resolution, spanning microelectrode array recordings from 20 arrays across 8 individuals with paralysis from spinal cord injury, amyotrophic lateral sclerosis or brainstem stroke, all enrolled in brain–computer interface clinical trials. These arrays broadly sample the crown of the precentral gyrus (PCG; thought to be composed largely of the premotor cortex (Brodmann area 6)). We found that body parts were highly intermixed, such that the entire body was represented in all sampled locations of the PCG, although the relative strength of body parts was roughly consistent with the motor homunculus17,18. We also found two speech-preferential areas with a broadly tuned, orofacial-dominant area in between them. Throughout the PCG, movement representations of the four limbs were interlinked, with homologous movements of different limbs (for example, toe curl and hand close) having correlated representations. These data provide evidence consistent with an intermixed, interrelated and behaviour-centred organization of the motor cortex3,21. The resulting map also provides important targeting information for brain–computer interfaces that seek to restore motor function.


  • Open Access

    Neural decoding of speech using deep neural ensembles

    by Seonghyun Yoon, Donald T. Avansino, Sasidhar Madugula, Alisa D. Levin, Chaofei Fan, Benyamin Abramovich Krasa, Akansha Singh, Christina Vo, Nick V. Hahn, Nicholas S. Card, et al.

    Speech brain-computer interfaces (BCIs) can restore rapid communication to people with paralysis, but decoding errors still limit performance. In recent brain-to-text decoding competitions, deep ensemble methods, which combine predictions from multiple independently trained decoders, have delivered striking accuracy improvements and account for the largest gains over baseline approaches. However, these methods have not previously been tested in real-time, require substantial computational resources, and their performance under various clinically relevant constraints remains poorly understood. Here, we present the first closed-loop test of deep ensembles in a participant with bilateral intracortical microelectrode arrays, demonstrating a reduction in word error rate from 33.7% to 26.0% on a large-vocabulary task. Using additional data from three participants, we then assess how these gains depend on baseline error rate, training dataset size, and ensemble size, including the resource-accuracy tradeoffs most relevant for real-world deployment. Finally, we introduce a computationally efficient pseudoensembling approach based on test-time augmentation that improves decoding accuracy while requiring only a single base decoder, greatly reducing the computational burden of ensembling. Together, these results show that the benefits of deep ensembling can be realized in real time and under practical resource constraints, bringing speech BCIs closer to broader clinical adoption.

  • Open Access

    Oscillatory activity in rostral middle frontal gyrus and subthalamic nucleus encode proactive inhibition in cortico-subcortical motor control network

    by Van Tuan Kiet Duong, Seyyed Bahram Borgheai, Enrico Opri, Faical Isbaine, Nicole C. Swann, Nicholas Au Yong, Svjetlana Miocinovic

    The current model of the action inhibition network includes the prefrontal cortex and the subthalamic nucleus (STN) connected via the prefrontal hyperdirect pathway. Proactive inhibition refers to preparatory mechanisms that facilitate action inhibition (i.e. enables a person to act with restraint), while reactive inhibition is a sudden stopping triggered by an external stimulus. Most research has focused on the reactive paradigm, with more limited investigation of proactive inhibition. We studied electrophysiologic activity in multiple cortical and STN regions in 17 patients with Parkinson′s disease using high-resolution intracranial electrodes. Subjects performed a Go/NoGo task and a simpler Go task. Proactive inhibition was assessed by contrasting Go trials in the context of two tasks. In the rostral middle frontal gyrus, we found increased beta oscillations during movement preparation in Go trials of the Go/NoGo task compared to the Go task. A similar but weaker, preparatory beta modulation was observed in dorsal STN, while central STN was associated with significant modulation in theta power prior to movement. We interpret this activity as a reflection of the role of these regions in proactively restraining anticipated responses. Conversely, inferior frontal gyrus and ventral STN were primarily engaged during rapid post–cue action control. Specifically, withholding of action after the NoGo signal was accompanied by increased theta activity in these regions. Beta modulation within the STN mirrored those of sensorimotor cortex during successful inhibition and movement execution. In both regions, beta activity decreased during movement and was higher when movement was withheld. We conclude that communication within the hypothesized motor control network is frequency dependent, with key nodes promoting specific functions.

  • Top Cited Papers
    Open Access

    Encoding of speech modes and loudness in ventral precentral gyrus

    by Aparna Srinivasan, Maitreyee Wairagkar, Carrina Iacobacci, Xianda Hou, Nicholas S. Card, Brandon G. Jacques, Anna L. Pritchard, Payton H. Bechefsky, Leigh R. Hochberg, Nicholas AuYong, et al.

    The ability to vary the mode and loudness of speech is an important part of the expressive range of human vocal communication. However, the encoding of these behaviors in the ventral precentral gyrus (vPCG) has not been studied at the resolution of neuronal firing rates. We investigated this in two participants who had intracortical microelectrode arrays implanted in their vPCG as part of a speech neuroprosthesis clinical trial. Neuronal firing rates modulated strongly in vPCG as a function of attempted mimed, whispered, normal or loud speech. At the neural ensemble level, mode/loudness and phonemic content were encoded in distinct neural subspaces. Attempted mode/loudness could be decoded from vPCG with 94% and 89% accuracy for the two participants, and corresponding neural preparatory activity at 640 ms and 270 ms before speech onset enabled 80% decoding accuracy, respectively. We then developed a closed-loop loudness decoder that achieved 94% online accuracy in modulating a brain-to-text speech neuroprosthesis output based on attempted loudness. These findings demonstrate the feasibility of decoding mode and loudness from vPCG, paving the way for speech neuroprostheses capable of synthesizing more expressive speech.


  • Open Access

    Cross-brain transfer of high-performance intracortical speech and handwriting BCIs

    by Alisa D. Levin, Donald T. Avansino, Foram B. Kamdar, Nicholas S. Card, Maitreyee Wairagkar, Brandon G. Jacques, Justin J. Jude, Carrina Iacobacci, Bayardo E. Lacayo, Payton H. Bechefsky, et al.

    Intracortical brain-computer interfaces (BCIs) that decode complex movements, such as handwriting and speech, can require substantial training data to achieve high performance. We investigated whether leveraging the neural activity recordings of previous users could reduce this initial data collection burden for new BCI users (an approach we call “cross-brain transfer”). Using intracortical recordings from five BrainGate2 clinical trial participants, we tested cross-brain transfer for both speech and handwriting neural decoders trained and evaluated on general, unconstrained corpora of spoken and written English. We found that cross-brain transfer improved decoding performance when training data from the target user was limited (< 200 sentences), and that dataset-specific input layers to the decoder were critical for combining data across users. Without trainable input layers, transfer failed and performed worse than training from scratch on target user data only. Finally, we measured the effectiveness of cross-brain transfer relative to training with (1) more data from the same user and (2) more electrode-permuted data from the same user, which simulates sampling from another brain with identical neural latent structure. In some cases (T16 speech, T12 handwriting), cross-brain transfer appeared as effective as additional permuted data from the same user, while in others (T12 speech, T15 speech) electrode-permuted data was more beneficial. Our results successfully demonstrate and characterize cross-brain transfer learning between multiple intracortical BCI users, for both speech and handwriting, using a general open-ended dataset not restricted to small sets of words or phrases. This work highlights a promising path towards addressing a key barrier to the clinical translation of BCIs, while clarifying when cross-brain transfer may be most beneficial and the decoder design choices needed to realize those gains.


  • Open Access

    Spinal interneuron population dynamics underlying flexible pattern generation

    by Lahiru N. Wimalasena, Chethan Pandarinath, Nicholas AuYong

    The mammalian spinal locomotor network is composed of diverse populations of interneurons that collectively orchestrate and execute a range of locomotor behaviors. As the number of identified classes of spinal interneurons constituting the locomotor network continues to grow, it still remains unclear how the network’s collective activity corresponds to locomotor output on a step-by-step basis. To investigate this, we analyzed lumbar interneuron population recordings and multi-muscle electromyography from spinalized female cats performing air stepping and used artificial intelligence methods to uncover state space trajectories of spinal interneuron population activity on single step cycles and at millisecond timescales. Our analyses of interneuron population trajectories revealed that traversal of specific state space regions held millisecond-timescale correspondence to the timing adjustments of extensor-flexor alternation. Similarly, we found that small variations in the path of state space trajectories were tightly linked to single-step, microvolt-scale adjustments in the magnitude of muscle output. These results reveal a previously-unseen regional organization of spinal interneuron state space, which may serve as a unifying framework to study spinal network function across the diversity of behaviors the spinal cord is capable of producing.


  • Open Access

    Preparatory encoding of diverse features of intended movement in the human motor cortex

    by Mattia Rigotti-Thompson, Samuel R. Nason-Tomaszewski, Payton Bechefsky, Alexander Acosta, Nick Hahn, Donald Avansino, Brice Richards, Claire Nicolas, Yahia H. Ali, Jaimie M. Henderson, et al.

    Over the course of a voluntary movement, motor cortical activity exhibits a transition from preparation to execution, with markedly different activity across these phases. Preparatory activity in particular might be used to improve brain-computer interfaces (BCIs) that harness brain activity to control external assistive devices, for example by anticipating a user’s intended movement trajectory for quick and fluid performance. However, to leverage preparatory activity for clinical BCIs, we must first understand which features of upcoming movements are encoded by preparatory activity in humans. In this work, we collected intracortical recordings from 3 research participants in the BrainGate2 clinical trial to investigate whether diverse features of movement, such as direction, curvature, and distance, are encoded by preparatory activity in the human motor cortex. We first show that preparatory activity is tuned to the direction of upcoming movements, and this tuning is largely preserved across movements with different effectors. Further investigation demonstrated this preparatory activity is also informative of initial and endpoint directions of curved movement trajectories, and encodes movement distance and speed independently. Finally, we present an online control paradigm that leverages preparatory activity to predict movements towards intended directions in advance, yielding rapid, self-paced control of a computer cursor by human participants. Altogether, these results demonstrate that preparatory activity in the human motor cortex encodes rich features of upcoming movement, highlighting its potential use for high performance brain-computer interface applications.

  • Open Access

    Bilateral Deep Brain Stimulation of the Subthalamic Nucleus or Globus Pallidus Internus Improves Gait Impairment in Parkinson’s Disease

    by Jamal Al Ali, J. Lucas Mckay, Joe R. Nocera, Faical Isbaine, Julie T. Tran, Paola Testini, Shirley D. Triche, Christine D. Esper, Pratibha Aia, Laura M. Scorr, et al.

    Background and objectives: The effect of deep brain stimulation (DBS) on gait in patients with Parkinson’s disease (PD) is variable but prospective, long-term studies—especially including globus pallidus interna (GPi) stimulation—remain sparse. We tested the hypothesis that subthalamic nucleus (STN) and GPi DBS exert acute and chronic effects on gait in patients with PD. Methods: Gait kinematics were collected prospectively on patients with PD with bilateral or unilateral STN or GPi DBS, at baseline before initial DBS activation (n=104), acutely after activation (n=102), and chronically at 1-month (n=75) and 12-months (n=82). Gait speed was the main outcome measure. Kinematic measures of pace, rhythm, and variability and clinical scales were secondary outcome measures. Results: The average gait speed at baseline in levodopa off state was abnormally slow (80.6 cm/s). DBS activation acutely increased gait speed (97.4 cm/s, p < 0.001) which was maintained at 1- (93.0 cm/s, p < 0.001) and 12-month follow up (91.2 cm/s, p < 0.001). Acute changes during initial programming predicted chronic gait outcomes. When DBS targets were analyzed separately, only patients receiving bilateral STN or GPi DBS improved gait speed along with kinematic measures of pace, rhythm, and variability. Preoperative levodopa response of the MDS-UPDRS axial sub-score correlated with gait speed response to DBS while the total score did not. Discussion: Patients who received either bilateral STN or GPi DBS improved gait kinematic in levodopa off state at 1 year so both targets may be considered for treatment of gait dysfunction, unlike unilateral implantations which resulted in no change. Acute effects of STN or GPi DBS on gait speed should be considered when programming patients as they predict chronic outcomes. When determining DBS candidacy, the degree of axial symptom improvement with levodopa should be considered.