Large Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech
Authors:
- Corentin PuffayCPCorentin Puffay
- Gavin MischlerGMGavin Mischler
- Vishal ChoudhariVCVishal Choudhari
- Jonas VanthornhoutJVJonas Vanthornhout
- Stephan BickelSBStephan Bickel
- Ashesh D. MehtaAMAshesh D. Mehta
- Catherine SchevonCSCatherine Schevon
- Guy M. McKhannGMGuy M. McKhann
- Hugo Van HammeHHHugo Van Hamme
- Tom FrancartTFTom Francart
- Nima MesgaraniNMNima Mesgarani
Preprint
Abstract
Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brain’s contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation “cocktail party” paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.Keywords
All Related Versions
- Published version: Imaging Neuroscience, 4.
Funding Information
- Research Foundation - Flanders (1S49823N)
- Research Foundation - Flanders (1290821N)
- National Institutes of Health
- National Institute on Deafness and Other Communication Disorders
- Marie-Josee and Henry R. Kravis Foundation
This publication has 31 references indexed in Scilit:
- Contextual feature extraction hierarchies converge in large language models and the brainNature Machine Intelligence, 2024
- Robust neural tracking of linguistic speech representations using a convolutional neural networkJournal of Neural Engineering, 2023
- Evidence of a predictive coding hierarchy in the human brain listening to speechNature Human Behaviour, 2023
- Shared computational principles for language processing in humans and deep language modelsNature Neuroscience, 2022
- Brains and algorithms partially converge in natural language processingCommunications Biology, 2022
- The neural architecture of language: Integrative modeling converges on predictive processingProceedings of the National Academy of Sciences, 2021
- Topographic specificity of alpha power during auditory spatial attentionNeuroImage, 2020
- Hierarchical Encoding of Attended Auditory Objects in Multi-talker Speech PerceptionNeuron, 2019
- Rapid Transformation from Auditory to Linguistic Representations of Continuous SpeechCurrent Biology, 2018
- Electrophysiological Correlates of Semantic Dissimilarity Reflect the Comprehension of Natural, Narrative SpeechCurrent Biology, 2018