Amirhossein Khalilian-Gourtani
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Amirhossein Khalilian-Gourtani has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
- Adeen Flinker
Department of Neurology, NYU Langone Medical Center, New York, USA
9 shared publications - Orrin Devinsky
Comprehensive Epilepsy Center NYU Langone Medical Center New York New York USA
6 shared publications Daniel Friedman
Department of Neurology, New York University, New York, USA
5 shared publications- Patricia Dugan
New York University School of Medicine
5 shared publications - Yao Wang
New York University Tandon School of Engineering, Electrical and Computer Engineering Department, Brooklyn, NY, USA
5 shared publications Yao Wang
Department of Electrical and Computer Engineering, New York University, New York, USA
4 shared publications
Latest Publications
- Open Access
Prediction of Stimulation-Defined Eloquent Cortex Using Graph-Theoretical Connectivity from Electrocorticography During Presurgical Mapping
Background and Objectives: Electrical stimulation mapping (ESM) is the clinical gold standard for identifying eloquent cortex during presurgical evaluation but is time-intensive, constrained by incomplete cortical sampling, and limited by patient tolerance. We investigated whether features derived from electrocorticography (ECoG), including functional connectivity measures, can provide complementary information for identifying functionally critical cortex and examined how predictive performance varies across functional domains, behavioral tasks, and data quantity. Methods: Fourteen patients undergoing intracranial monitoring for epilepsy surgery performed speech production tasks while ECoG was recorded independently of stimulation mapping. Graph-theoretic functional connectivity features derived from high-gamma activity (70–150 Hz), combined with anatomical region encoding, were used to train machine learning classifiers, and predictive performance was evaluated using leave-one-subject-out validation with ESM-defined functional deficits used as ground-truth outcomes. Results: Fourteen patients were included. Performance differed across functional domains, with motor-critical electrodes identified with the highest accuracy (ROC–AUC 0.929 ± 0.061; PR–AUC 0.755 ± 0.191), followed by speech arrest (ROC–AUC 0.793 ± 0.103; PR–AUC 0.550 ± 0.196), whereas language-critical electrodes were more difficult to robustly predict (ROC–AUC 0.761 ± 0.160; PR–AUC 0.385 ± 0.167). Additional signal-derived features provided limited benefit beyond anatomical and connectivity features. Performance also varied across tasks and with the number of available trials, with motor prediction remaining stable across trial counts and speech arrest prediction improving with increasing trial counts up to approximately 10 trials before plateauing. In addition, evaluation at the level of stimulation pairs increased sensitivity, with only modest changes in overall performance. Discussion: Graph-theoretic connectivity analysis of ECoG provides complementary information for identifying stimulation-defined functional criticality and supports presurgical mapping. Differences in predictive performance across functional domains likely reflect underlying neuro-physiologic organization, with connectivity providing the clearest improvement for speech arrest prediction. Combined-label prediction strategies increased sensitivity at the expense of specificity, reflecting a clinically relevant tradeoff between broad detection of critical cortex and precise localization. These findings suggest that connectivity-informed approaches may help guide task selection and improve mapping efficiency while complementing electrical stimulation mapping.
Open AccessPrediction of Stimulation-Defined Eloquent Cortex Using Graph-Theoretical Connectivity from Electrocorticography During Presurgical Mapping
Background and Objectives: Electrical stimulation mapping (ESM) is the clinical gold standard for identifying eloquent cortex during presurgical evaluation but is time-intensive, constrained by incomplete cortical sampling, and limited by patient tolerance. We investigated whether features derived from electrocorticography (ECoG), including functional connectivity measures, can provide complementary information for identifying functionally critical cortex and examined how predictive performance varies across functional domains, behavioral tasks, and data quantity. Methods: Fourteen patients undergoing intracranial monitoring for epilepsy surgery performed speech production tasks while ECoG was recorded independently of stimulation mapping. Graph-theoretic functional connectivity features derived from high-gamma activity (70–150 Hz), combined with anatomical region encoding, were used to train machine learning classifiers, and predictive performance was evaluated using leave-one-subject-out validation with ESM-defined functional deficits used as ground-truth outcomes. Results: Fourteen patients were included. Performance differed across functional domains, with motor-critical electrodes identified with the highest accuracy (ROC–AUC 0.929 ± 0.061; PR–AUC 0.755 ± 0.191), followed by speech arrest (ROC–AUC 0.793 ± 0.103; PR–AUC 0.550 ± 0.196), whereas language-critical electrodes were more difficult to robustly predict (ROC–AUC 0.761 ± 0.160; PR–AUC 0.385 ± 0.167). Additional signal-derived features provided limited benefit beyond anatomical and connectivity features. Performance also varied across tasks and with the number of available trials, with motor prediction remaining stable across trial counts and speech arrest prediction improving with increasing trial counts up to approximately 10 trials before plateauing. In addition, evaluation at the level of stimulation pairs increased sensitivity, with only modest changes in overall performance. Discussion: Graph-theoretic connectivity analysis of ECoG provides complementary information for identifying stimulation-defined functional criticality and supports presurgical mapping. Differences in predictive performance across functional domains likely reflect underlying neuro-physiologic organization, with connectivity providing the clearest improvement for speech arrest prediction. Combined-label prediction strategies increased sensitivity at the expense of specificity, reflecting a clinically relevant tradeoff between broad detection of critical cortex and precise localization. These findings suggest that connectivity-informed approaches may help guide task selection and improve mapping efficiency while complementing electrical stimulation mapping.
- Open Access
Neural Dynamics of Automatic Speech Production
Speech is a defining human behavior, and this ability depends critically on speech motor cortex. While the ventral precentral and postcentral gyri are classically regarded as chiefly articulatory and somatosensory regions, a growing body of literature challenges this simplification. Most prior research, however, has examined cued or structured speech production tasks, neglecting the automatic, overlearned speech commonly utilized in clinical assessment. Consequently, the neural dynamics and precise timing of cortical recruitment during automatic speech remain poorly understood. Here, we present intracranial electrocorticography (ECoG) recordings from the left perisylvian cortex in participants performing automatic speech such as counting and recitation of overlearned sequences. We investigate neural dynamics using encoding (multivariate temporal response function) and decoding (deep neural network speech synthesis) models. We show that automatic speech engages a distributed network across superior temporal, precentral, and post-central cortices, characterized by attenuated pre-articulatory activity and weaker frontal encoding. Furthermore, two complementary decoding strategies reveal that speech motor cortex represents a mixture of feedforward and feedback signals, with a subset of sites exhibiting exclusively feed-forward dynamics. These results delineate the spatiotemporal cortical organization of automatic speech and establish that the speech motor cortex supports more complex dynamics than purely feedforward control.
Open AccessNeural Dynamics of Automatic Speech Production
Speech is a defining human behavior, and this ability depends critically on speech motor cortex. While the ventral precentral and postcentral gyri are classically regarded as chiefly articulatory and somatosensory regions, a growing body of literature challenges this simplification. Most prior research, however, has examined cued or structured speech production tasks, neglecting the automatic, overlearned speech commonly utilized in clinical assessment. Consequently, the neural dynamics and precise timing of cortical recruitment during automatic speech remain poorly understood. Here, we present intracranial electrocorticography (ECoG) recordings from the left perisylvian cortex in participants performing automatic speech such as counting and recitation of overlearned sequences. We investigate neural dynamics using encoding (multivariate temporal response function) and decoding (deep neural network speech synthesis) models. We show that automatic speech engages a distributed network across superior temporal, precentral, and post-central cortices, characterized by attenuated pre-articulatory activity and weaker frontal encoding. Furthermore, two complementary decoding strategies reveal that speech motor cortex represents a mixture of feedforward and feedback signals, with a subset of sites exhibiting exclusively feed-forward dynamics. These results delineate the spatiotemporal cortical organization of automatic speech and establish that the speech motor cortex supports more complex dynamics than purely feedforward control.
- Open Access
- 4 February 2026
- Published by eLife Sciences Publications, Ltd in eLife preprints
Decoding state specific connectivity during speech production and perception
Understanding how dynamic brain networks support language perception and production is central to cognitive neuroscience. A vast network based literature has employed functional connectivity (FC), primarily using resting-state and task-based fMRI. However, methodological limitations have hindered this approach in language processing, particularly during speech production. Here, we address this gap by employing a large cohort of electrocorticographic (ECoG) patients (N=42) to investigate the networks driving speech perception and production. We acquired data while patients were engaged in a controlled battery of speech production tasks focusing on five cognitive states (auditory perception, picture perception, reading perception, speech production, and baseline). Using linear classifiers we were able to robustly decode cognitive states from single-trial FC (i.e. Pearson correlations) of the neural activity patterns, achieving a mean accuracy of 64.4%. These classifiers revealed distinct network signatures underlying auditory and visual perception as well as speech production via stable network connectivity. Importantly, the network signatures included both regions with robust local neural activity and those with minimal or no detectable activation. Such signatures indicate that even low-activity regions contribute critically to differentiating cognitive states. Our findings underscore the significance of functional connectivity analysis as a complementary dimension to investigating local neural activity, and suggest that the functional networks supporting speech extend beyond the most metabolically active regions.
- 4 February 2026
- Published by eLife Sciences Publications, Ltd in eLife preprints
Open AccessDecoding state specific connectivity during speech production and perception
Understanding how dynamic brain networks support language perception and production is central to cognitive neuroscience. A vast network based literature has employed functional connectivity (FC), primarily using resting-state and task-based fMRI. However, methodological limitations have hindered this approach in language processing, particularly during speech production. Here, we address this gap by employing a large cohort of electrocorticographic (ECoG) patients (N=42) to investigate the networks driving speech perception and production. We acquired data while patients were engaged in a controlled battery of speech production tasks focusing on five cognitive states (auditory perception, picture perception, reading perception, speech production, and baseline). Using linear classifiers we were able to robustly decode cognitive states from single-trial FC (i.e. Pearson correlations) of the neural activity patterns, achieving a mean accuracy of 64.4%. These classifiers revealed distinct network signatures underlying auditory and visual perception as well as speech production via stable network connectivity. Importantly, the network signatures included both regions with robust local neural activity and those with minimal or no detectable activation. Such signatures indicate that even low-activity regions contribute critically to differentiating cognitive states. Our findings underscore the significance of functional connectivity analysis as a complementary dimension to investigating local neural activity, and suggest that the functional networks supporting speech extend beyond the most metabolically active regions.
- Open Access
Decoding state specific connectivity during speech production and perception
Understanding how dynamic brain networks support language perception and production is central to cognitive neuroscience. A vast network based literature has employed functional connectivity (FC), primarily using resting-state and task-based fMRI. However, methodological limitations have hindered this approach in language processing, particularly during speech production. Here, we address this gap by employing a large cohort of electrocorticographic (ECoG) patients (N=42) to investigate the networks driving speech perception and production. We acquired data while patients were engaged in a controlled battery of speech production tasks focusing on five cognitive states (auditory perception, picture perception, reading perception, speech production, and baseline). Using linear classifiers we were able to robustly decode cognitive states from single-trial FC (i.e. Pearson correlations) of the neural activity patterns, achieving a mean accuracy of 64.4%. These classifiers revealed distinct network signatures underlying auditory and visual perception as well as speech production via stable network connectivity. Importantly, the network signatures included both regions with robust local neural activity and those with minimal or no detectable activation. Such signatures indicate that even low-activity regions contribute critically to differentiating cognitive states. Our findings underscore the significance of functional connectivity analysis as a complementary dimension to investigating local neural activity, and suggest that the functional networks supporting speech extend beyond the most metabolically active regions.
Open AccessDecoding state specific connectivity during speech production and perception
Understanding how dynamic brain networks support language perception and production is central to cognitive neuroscience. A vast network based literature has employed functional connectivity (FC), primarily using resting-state and task-based fMRI. However, methodological limitations have hindered this approach in language processing, particularly during speech production. Here, we address this gap by employing a large cohort of electrocorticographic (ECoG) patients (N=42) to investigate the networks driving speech perception and production. We acquired data while patients were engaged in a controlled battery of speech production tasks focusing on five cognitive states (auditory perception, picture perception, reading perception, speech production, and baseline). Using linear classifiers we were able to robustly decode cognitive states from single-trial FC (i.e. Pearson correlations) of the neural activity patterns, achieving a mean accuracy of 64.4%. These classifiers revealed distinct network signatures underlying auditory and visual perception as well as speech production via stable network connectivity. Importantly, the network signatures included both regions with robust local neural activity and those with minimal or no detectable activation. Such signatures indicate that even low-activity regions contribute critically to differentiating cognitive states. Our findings underscore the significance of functional connectivity analysis as a complementary dimension to investigating local neural activity, and suggest that the functional networks supporting speech extend beyond the most metabolically active regions.
- Open Access
A probabilistic functional atlas based on extraoperative electrocortical stimulation mapping
Background: Direct electrocortical stimulation (DES) is the clinical gold standard for identifying eloquent cortex and guiding neurosurgical intervention, yet prior intraoperative DES during awake craniotomies have been limited by intraoperative sampling constraints and density-based rather than probabilistic analyses. The probability of typical and atypical cortical organization has not been fully explored, especially in epilepsy populations. We sought to generate a probabilistic atlas of motor, sensory, and language functions using extraoperative DES in a large cohort of patients with epilepsy. Methods: We retrospectively analyzed 2,124 extraoperative DES trials from 125 patients undergoing intracranial monitoring (2008–2023). Positive and negative trials were mapped to Montreal Neurological Institute space, parcellated with the Human Connectome Project atlas, and analyzed using probability mapping, bootstrapped region-of-interest hit probabilities, hierarchical clustering, and kernel density estimation. Mixed-effects models assessed clinical predictors of language disruption. Results: Probabilistic maps revealed regions of increased likelihood for eliciting functional responses in expected sensorimotor and language territories, but also demonstrated marked variability and deviations from expected cortical locations. Language disruption occurred in 338 trials, motor in 520, and sensory in 370. Instead of observing high probabilities and low inter-patient variability isolated to classic perisylvian locations (e.g., Broca’s and Wernicke’s areas), the likelihood of language disruption followed graded probabilistic gradients with high inter-patient variability. The middle frontal gyrus emerged as a consistent locus of naming and speech arrest. Motor phenomena extended into parietal association cortex. Higher-order experiences, including forced thoughts and feelings of presence, were reproducibly evoked from frontal and temporoparietal sites. Early seizure onset and temporal lobe lesions predicted lower naming disruption probabilities. Conclusions: This extraoperative DES atlas, the largest to date, demonstrates that eloquent cortical functions are organized along probabilistic continua rather than fixed regions. Findings highlight the middle frontal gyrus as a critical language node, extend motor mapping into parietal cortex, and delineate reproducible experiential phenomena. Substantial inter-patient variability underscores the necessity of individualized mapping in surgical planning.
Open AccessA probabilistic functional atlas based on extraoperative electrocortical stimulation mapping
Background: Direct electrocortical stimulation (DES) is the clinical gold standard for identifying eloquent cortex and guiding neurosurgical intervention, yet prior intraoperative DES during awake craniotomies have been limited by intraoperative sampling constraints and density-based rather than probabilistic analyses. The probability of typical and atypical cortical organization has not been fully explored, especially in epilepsy populations. We sought to generate a probabilistic atlas of motor, sensory, and language functions using extraoperative DES in a large cohort of patients with epilepsy. Methods: We retrospectively analyzed 2,124 extraoperative DES trials from 125 patients undergoing intracranial monitoring (2008–2023). Positive and negative trials were mapped to Montreal Neurological Institute space, parcellated with the Human Connectome Project atlas, and analyzed using probability mapping, bootstrapped region-of-interest hit probabilities, hierarchical clustering, and kernel density estimation. Mixed-effects models assessed clinical predictors of language disruption. Results: Probabilistic maps revealed regions of increased likelihood for eliciting functional responses in expected sensorimotor and language territories, but also demonstrated marked variability and deviations from expected cortical locations. Language disruption occurred in 338 trials, motor in 520, and sensory in 370. Instead of observing high probabilities and low inter-patient variability isolated to classic perisylvian locations (e.g., Broca’s and Wernicke’s areas), the likelihood of language disruption followed graded probabilistic gradients with high inter-patient variability. The middle frontal gyrus emerged as a consistent locus of naming and speech arrest. Motor phenomena extended into parietal association cortex. Higher-order experiences, including forced thoughts and feelings of presence, were reproducibly evoked from frontal and temporoparietal sites. Early seizure onset and temporal lobe lesions predicted lower naming disruption probabilities. Conclusions: This extraoperative DES atlas, the largest to date, demonstrates that eloquent cortical functions are organized along probabilistic continua rather than fixed regions. Findings highlight the middle frontal gyrus as a critical language node, extend motor mapping into parietal cortex, and delineate reproducible experiential phenomena. Substantial inter-patient variability underscores the necessity of individualized mapping in surgical planning.
- Open Access
Frontal cortex organization supporting audiovisual processing during naturalistic viewing
: Our brains dynamically adapt to a multisensory world by orchestrating diverse inputs across sensory streams. This process engages multiple brain regions, but it remains unclear how audiovisual stimuli are represented and evolve over time, especially in naturalistic scenarios. Here, we employed a movie-viewing paradigm to explore this question. We recorded intracranial electrocorticography (iEEG) to measure brain activity in 19 participants watching a short multilingual movie. Using unsupervised clustering and supervised encoding models, we identified a robust modality-specific gradient in the frontal cortex, wherein the ventral division primarily processes auditory information and the dorsal division processes visual inputs. Further, we found that this cortical organization dynamically changed, adapting to different movie contexts. This result potentially reflects flexible audiovisual-resource assignment to construct a coherent percept of the movie. Leveraging behavioral ratings, we found that the frontal cortex is the primary site in this modality assignment process. Together, our findings shed new light on the functional architecture of the frontal cortex underlying flexible multisensory representation and integration in natural contexts.
Open AccessFrontal cortex organization supporting audiovisual processing during naturalistic viewing
: Our brains dynamically adapt to a multisensory world by orchestrating diverse inputs across sensory streams. This process engages multiple brain regions, but it remains unclear how audiovisual stimuli are represented and evolve over time, especially in naturalistic scenarios. Here, we employed a movie-viewing paradigm to explore this question. We recorded intracranial electrocorticography (iEEG) to measure brain activity in 19 participants watching a short multilingual movie. Using unsupervised clustering and supervised encoding models, we identified a robust modality-specific gradient in the frontal cortex, wherein the ventral division primarily processes auditory information and the dorsal division processes visual inputs. Further, we found that this cortical organization dynamically changed, adapting to different movie contexts. This result potentially reflects flexible audiovisual-resource assignment to construct a coherent percept of the movie. Leveraging behavioral ratings, we found that the frontal cortex is the primary site in this modality assignment process. Together, our findings shed new light on the functional architecture of the frontal cortex underlying flexible multisensory representation and integration in natural contexts.
- 5 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 11 (25730436) , 201-212
GroupCDL: Interpretable Denoising and Compressed Sensing MRI via Learned Group-Sparsity and Circulant Attention
Nonlocal self-similarity within images has become an increasingly popular prior in deep-learning models. Despite their successful image restoration performance, such models remain largely uninterpretable due to their black-box construction. Our previous studies have shown that interpretable construction of a fully convolutional denoiser (CDLNet), with performance on par with state-of-the-art black-box counterparts, is achievable by unrolling a convolutional dictionary learning algorithm. In this manuscript, we seek an interpretable construction of a convolutional network with a nonlocal self-similarity prior that performs on par with black-box nonlocal models. We show that such an architecture can be effectively achieved by upgrading the $\ell _{1}$ sparsity prior (soft-thresholding) of CDLNet to an image-adaptive group-sparsity prior (group-thresholding). The proposed learned group-thresholding makes use of nonlocal attention to perform spatially varying soft-thresholding on the latent representation. To enable effective training and inference on large images with global artifacts, we propose a novel circulant-sparse attention. We achieve competitive natural-image denoising performance compared to black-box nonlocal DNNs and transformers. The interpretable construction of our network allows for a straightforward extension to Compressed Sensing MRI (CS-MRI), yielding state-of-the-art performance. Lastly, we show robustness to noise-level mismatches between training and inference for denoising and CS-MRI reconstruction.
- 5 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 11 (25730436) , 201-212
GroupCDL: Interpretable Denoising and Compressed Sensing MRI via Learned Group-Sparsity and Circulant Attention
Nonlocal self-similarity within images has become an increasingly popular prior in deep-learning models. Despite their successful image restoration performance, such models remain largely uninterpretable due to their black-box construction. Our previous studies have shown that interpretable construction of a fully convolutional denoiser (CDLNet), with performance on par with state-of-the-art black-box counterparts, is achievable by unrolling a convolutional dictionary learning algorithm. In this manuscript, we seek an interpretable construction of a convolutional network with a nonlocal self-similarity prior that performs on par with black-box nonlocal models. We show that such an architecture can be effectively achieved by upgrading the $\ell _{1}$ sparsity prior (soft-thresholding) of CDLNet to an image-adaptive group-sparsity prior (group-thresholding). The proposed learned group-thresholding makes use of nonlocal attention to perform spatially varying soft-thresholding on the latent representation. To enable effective training and inference on large images with global artifacts, we propose a novel circulant-sparse attention. We achieve competitive natural-image denoising performance compared to black-box nonlocal DNNs and transformers. The interpretable construction of our network allows for a straightforward extension to Compressed Sensing MRI (CS-MRI), yielding state-of-the-art performance. Lastly, we show robustness to noise-level mismatches between training and inference for denoising and CS-MRI reconstruction.
- Open Access
- 3 December 2024
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 121 (50)
A corollary discharge circuit in human speech
When we vocalize, our brain distinguishes self-generated sounds from external ones. A corollary discharge signal supports this function in animals; however, in humans, its exact origin and temporal dynamics remain unknown. We report electrocorticographic recordings in neurosurgical patients and a connectivity analysis framework based on Granger causality that reveals major neural communications. We find a reproducible source for corollary discharge across multiple speech production paradigms localized to the ventral speech motor cortex before speech articulation. The uncovered discharge predicts the degree of auditory cortex suppression during speech, its well-documented consequence. These results reveal the human corollary discharge source and timing with far-reaching implication for speech motor-control as well as auditory hallucinations in human psychosis.
- 3 December 2024
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 121 (50)
Open AccessA corollary discharge circuit in human speech
When we vocalize, our brain distinguishes self-generated sounds from external ones. A corollary discharge signal supports this function in animals; however, in humans, its exact origin and temporal dynamics remain unknown. We report electrocorticographic recordings in neurosurgical patients and a connectivity analysis framework based on Granger causality that reveals major neural communications. We find a reproducible source for corollary discharge across multiple speech production paradigms localized to the ventral speech motor cortex before speech articulation. The uncovered discharge predicts the degree of auditory cortex suppression during speech, its well-documented consequence. These results reveal the human corollary discharge source and timing with far-reaching implication for speech motor-control as well as auditory hallucinations in human psychosis.
- Top Cited PapersOpen Access
- 8 April 2024
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 6 (4) , 467-480
A neural speech decoding framework leveraging deep learning and speech synthesis
Decoding human speech from neural signals is essential for brain–computer interface (BCI) technologies that aim to restore speech in populations with neurological deficits. However, it remains a highly challenging task, compounded by the scarce availability of neural signals with corresponding speech, data complexity and high dimensionality. Here we present a novel deep learning-based neural speech decoding framework that includes an ECoG decoder that translates electrocorticographic (ECoG) signals from the cortex into interpretable speech parameters and a novel differentiable speech synthesizer that maps speech parameters to spectrograms. We have developed a companion speech-to-speech auto-encoder consisting of a speech encoder and the same speech synthesizer to generate reference speech parameters to facilitate the ECoG decoder training. This framework generates natural-sounding speech and is highly reproducible across a cohort of 48 participants. Our experimental results show that our models can decode speech with high correlation, even when limited to only causal operations, which is necessary for adoption by real-time neural prostheses. Finally, we successfully decode speech in participants with either left or right hemisphere coverage, which could lead to speech prostheses in patients with deficits resulting from left hemisphere damage.
- 8 April 2024
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 6 (4) , 467-480
Top Cited PapersOpen AccessA neural speech decoding framework leveraging deep learning and speech synthesis
Decoding human speech from neural signals is essential for brain–computer interface (BCI) technologies that aim to restore speech in populations with neurological deficits. However, it remains a highly challenging task, compounded by the scarce availability of neural signals with corresponding speech, data complexity and high dimensionality. Here we present a novel deep learning-based neural speech decoding framework that includes an ECoG decoder that translates electrocorticographic (ECoG) signals from the cortex into interpretable speech parameters and a novel differentiable speech synthesizer that maps speech parameters to spectrograms. We have developed a companion speech-to-speech auto-encoder consisting of a speech encoder and the same speech synthesizer to generate reference speech parameters to facilitate the ECoG decoder training. This framework generates natural-sounding speech and is highly reproducible across a cohort of 48 participants. Our experimental results show that our models can decode speech with high correlation, even when limited to only causal operations, which is necessary for adoption by real-time neural prostheses. Finally, we successfully decode speech in participants with either left or right hemisphere coverage, which could lead to speech prostheses in patients with deficits resulting from left hemisphere damage.
- Open Access
- 5 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 3, 196-211
CDLNet: Noise-Adaptive Convolutional Dictionary Learning Network for Blind Denoising and Demosaicing
Deep learning based methods hold state-of-the-art results in low-level image processing tasks, but remain difficult to interpret due to their black-box construction. Unrolled optimization networks present an interpretable alternative to constructing deep neural networks by deriving their architecture from classical iterative optimization methods without use of tricks from the standard deep learning tool-box. So far, such methods have demonstrated performance close to that of state-of-the-art models while using their interpretable construction to achieve a comparably low learned parameter count. In this work, we propose an unrolled convolutional dictionary learning network (CDLNet) and demonstrate its competitive denoising and joint denoising and demosaicing (JDD) performance both in low and high parameter count regimes. Specifically, we show that the proposed model outperforms state-of-the-art fully convolutional denoising and JDD models when scaled to a similar parameter count. In addition, we leverage the model’s interpretable construction to propose a noise-adaptive parameterization of thresholds in the network that enables state-of-the-art blind denoising performance, and near-perfect generalization on noise-levels unseen during training. Furthermore, we show that such performance extends to the JDD task and unsupervised learning.
- 5 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 3, 196-211
Open AccessCDLNet: Noise-Adaptive Convolutional Dictionary Learning Network for Blind Denoising and Demosaicing
Deep learning based methods hold state-of-the-art results in low-level image processing tasks, but remain difficult to interpret due to their black-box construction. Unrolled optimization networks present an interpretable alternative to constructing deep neural networks by deriving their architecture from classical iterative optimization methods without use of tricks from the standard deep learning tool-box. So far, such methods have demonstrated performance close to that of state-of-the-art models while using their interpretable construction to achieve a comparably low learned parameter count. In this work, we propose an unrolled convolutional dictionary learning network (CDLNet) and demonstrate its competitive denoising and joint denoising and demosaicing (JDD) performance both in low and high parameter count regimes. Specifically, we show that the proposed model outperforms state-of-the-art fully convolutional denoising and JDD models when scaled to a similar parameter count. In addition, we leverage the model’s interpretable construction to propose a noise-adaptive parameterization of thresholds in the network that enables state-of-the-art blind denoising performance, and near-perfect generalization on noise-levels unseen during training. Furthermore, we show that such performance extends to the JDD task and unsupervised learning.