Vishal Choudhari
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Vishal Choudhari has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Nima Mesgarani
Department of Electrical Engineering, Columbia University, New York, USA
5 shared publications - Stephan Bickel
The Feinstein Institutes for Medical Research, Northwell Health, Manhasset, USA
5 shared publications - Ashesh D. Mehta
The Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
4 shared publications - Catherine A. Schevon
Department of Neurology, Columbia University Irving Medical Center, New York, New York, U.S.A.
4 shared publications - Guy M. McKhann
Department of Neurosurgery, Columbia University Irving Medical Center, New York, NY, USA
4 shared publications - Adeen Flinker
Department of Neurology, NYU Langone Medical Center, New York, USA
2 shared publications
Latest Publications
- Open Access
- 11 May 2026
- Published by Springer Nature in Nature Neuroscience
- Vol. 29 (10) , 1-11
Real-time brain-controlled selective hearing enhances speech perception in multi-talker environments
Understanding speech in noisy environments is difficult for many people, and current hearing aids often fail because they amplify all sounds rather than the talker of interest. Auditory attention decoding (AAD) offers a potential solution by using the listener’s brain signals to identify and enhance the attended speaker, but it has been unclear whether this can provide real-time perceptual benefits. Here we used high-resolution intracranial electroencephalography in patients undergoing neurosurgical procedures to implement a closed-loop system that achieves the decoding fidelity necessary to dynamically amplify the attended talker. Across multiple experiments, the system improved speech intelligibility, reduced listening effort and was consistently preferred by subjects. It also tracked both instructed and self-initiated attention shifts. By providing direct evidence that a real-time, brain-controlled hearing system can enhance perception, this work establishes a key performance benchmark for future auditory brain–computer interfaces and advances AAD from a theoretical concept to a validated solution for personalized assistive hearing.
- 11 May 2026
- Published by Springer Nature in Nature Neuroscience
- Vol. 29 (10) , 1-11
Open AccessReal-time brain-controlled selective hearing enhances speech perception in multi-talker environments
Understanding speech in noisy environments is difficult for many people, and current hearing aids often fail because they amplify all sounds rather than the talker of interest. Auditory attention decoding (AAD) offers a potential solution by using the listener’s brain signals to identify and enhance the attended speaker, but it has been unclear whether this can provide real-time perceptual benefits. Here we used high-resolution intracranial electroencephalography in patients undergoing neurosurgical procedures to implement a closed-loop system that achieves the decoding fidelity necessary to dynamically amplify the attended talker. Across multiple experiments, the system improved speech intelligibility, reduced listening effort and was consistently preferred by subjects. It also tracked both instructed and self-initiated attention shifts. By providing direct evidence that a real-time, brain-controlled hearing system can enhance perception, this work establishes a key performance benchmark for future auditory brain–computer interfaces and advances AAD from a theoretical concept to a validated solution for personalized assistive hearing.
- Open Access
Speaker Identity is Robustly Encoded in Spatial Patterns of Intracranial EEG for Attention Decoding
The human auditory cortex robustly tracks attended speech, yet it remains unclear if speaker identity is encoded in spatial patterns of neural activity independent of temporal dynamics. Here, we demonstrate that the identity of an attended speaker is reliably reflected in distinct, time-invariant spatial activation maps in human intracranial EEG (iEEG). Leveraging these "neural fingerprints", we developed a novel framework for Auditory Attention Decoding (AAD) that shifts from traditional temporal envelope tracking to spatial speaker identification. By decoupling the decoding of "who" is speaking from "when" they are speaking, our modular system achieves state-of-the-art speech extraction, particularly in short time windows (<2 seconds) where temporal models typically fail. Furthermore, we observed a reciprocal shift in neural activity during attentional switches, confirming that these spatial codes dynamically track listener intent. These findings establish that speaker identity is a robust, spatially distributed feature in the auditory cortex, offering a high-speed, complementary mechanism for neuro-steered hearing technologies.
Open AccessSpeaker Identity is Robustly Encoded in Spatial Patterns of Intracranial EEG for Attention Decoding
The human auditory cortex robustly tracks attended speech, yet it remains unclear if speaker identity is encoded in spatial patterns of neural activity independent of temporal dynamics. Here, we demonstrate that the identity of an attended speaker is reliably reflected in distinct, time-invariant spatial activation maps in human intracranial EEG (iEEG). Leveraging these "neural fingerprints", we developed a novel framework for Auditory Attention Decoding (AAD) that shifts from traditional temporal envelope tracking to spatial speaker identification. By decoupling the decoding of "who" is speaking from "when" they are speaking, our modular system achieves state-of-the-art speech extraction, particularly in short time windows (<2 seconds) where temporal models typically fail. Furthermore, we observed a reciprocal shift in neural activity during attentional switches, confirming that these spatial codes dynamically track listener intent. These findings establish that speaker identity is a robust, spatially distributed feature in the auditory cortex, offering a high-speed, complementary mechanism for neuro-steered hearing technologies.
- Open Access
Large Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech
Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brain’s contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation “cocktail party” paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.
Open AccessLarge Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech
Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brain’s contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation “cocktail party” paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.
- Open Access
- 9 September 2024
- Published by Wiley in Advanced Science
- Vol. 11 (41) , e2401379
Brain‐Controlled Augmented Hearing for Spatially Moving Conversations in Multi‐Talker Environments
Focusing on a specific conversation amidst multiple interfering talkers is challenging, especially for those with hearing loss. Brain‐controlled assistive hearing devices aim to alleviate this problem by enhancing the attended speech based on the listener's neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, a realistic AAD task that involves multiple talkers taking turns as they continuously move in space in background noise is presented. Invasive electroencephalography (iEEG) data are collected from three neurosurgical patients as they focused on one of the two moving conversations. An enhanced brain‐controlled assistive hearing system that combines AAD and a binaural speaker‐independent speech separation model is presented. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve AAD accuracy. Subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of this approach in real‐world scenarios and marks a significant step toward developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.
- 9 September 2024
- Published by Wiley in Advanced Science
- Vol. 11 (41) , e2401379
Open AccessBrain‐Controlled Augmented Hearing for Spatially Moving Conversations in Multi‐Talker Environments
Focusing on a specific conversation amidst multiple interfering talkers is challenging, especially for those with hearing loss. Brain‐controlled assistive hearing devices aim to alleviate this problem by enhancing the attended speech based on the listener's neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, a realistic AAD task that involves multiple talkers taking turns as they continuously move in space in background noise is presented. Invasive electroencephalography (iEEG) data are collected from three neurosurgical patients as they focused on one of the two moving conversations. An enhanced brain‐controlled assistive hearing system that combines AAD and a binaural speaker‐independent speech separation model is presented. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve AAD accuracy. Subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of this approach in real‐world scenarios and marks a significant step toward developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.
- Open Access
Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments
Focusing on a specific conversation amidst multiple interfering talkers presents a significant challenge, especially for the hearing-impaired. Brain-controlled assistive hearing devices aim to alleviate this problem by separating complex auditory scenes into distinct speech streams and enhancing the attended speech based on the listener’s neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, we present a realistic AAD task that mirrors the dynamic nature of acoustic settings. This task involves focusing on one of two concurrent conversations, with multiple talkers taking turns and moving continuously in space with background noise. Invasive electroencephalography (iEEG) data were collected from three neurosurgical patients as they focused on one of the two moving conversations. We propose an enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve auditory attention decoding accuracy. Our subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of our approach in real-world scenarios and marks a significant step towards developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences. TAKEAWAYS - Brain-controlled hearing device for scenarios with moving conversations in multi-talker settings, closely mimicking real-world listening environments - Developed a binaural speech separation model that separates speech of moving talkers while retaining their spatial locations, enhancing auditory perception and auditory attention decoding - Proposed system enhances speech intelligibility and reduces listening effort in realistic acoustic scenes
Open AccessBrain-controlled augmented hearing for spatially moving conversations in multi-talker environments
Focusing on a specific conversation amidst multiple interfering talkers presents a significant challenge, especially for the hearing-impaired. Brain-controlled assistive hearing devices aim to alleviate this problem by separating complex auditory scenes into distinct speech streams and enhancing the attended speech based on the listener’s neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, we present a realistic AAD task that mirrors the dynamic nature of acoustic settings. This task involves focusing on one of two concurrent conversations, with multiple talkers taking turns and moving continuously in space with background noise. Invasive electroencephalography (iEEG) data were collected from three neurosurgical patients as they focused on one of the two moving conversations. We propose an enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve auditory attention decoding accuracy. Our subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of our approach in real-world scenarios and marks a significant step towards developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences. TAKEAWAYS - Brain-controlled hearing device for scenarios with moving conversations in multi-talker settings, closely mimicking real-world listening environments - Developed a binaural speech separation model that separates speech of moving talkers while retaining their spatial locations, enhancing auditory perception and auditory attention decoding - Proposed system enhances speech intelligibility and reduces listening effort in realistic acoustic scenes