Samuel A. Nastase

Cortical resources are allocated into systems specialized for processing ecologically relevant features of the world. These features are typically studied in isolation using controlled experimental stimuli, making it difficult to assess the relative importance of different kinds of features that tend to overlap during natural vision. In the current study, we evaluated the relative contributions of action, agent, and scene features in predicting cortical activity while participants viewed a 1-h nature documentary. We tested four sets of model features: semantic vectors capturing the observed action, agent, and scene features derived from an annotation of the stimulus, as well as low-level visual motion-energy features. We used banded ridge regression to fit vertex-wise encoding models from the combination of all four sets of features. While each set of features predicted neural activity in the expected areas, the action features predicted more widespread activity. A variance partitioning analysis revealed that the action features captured the most unique variance in neural activity across a much larger extent of cortex than the agent or scene features. Our findings suggest that cortical activity during dynamic, natural vision is dominated by features for understanding the actions of others.

We effortlessly extract behaviorally relevant information from dynamic visual input to understand the actions of others. Here, we compare behavioral judgments of action meaning against other models capturing semantic and visual features. We measured brain activity using fMRI while participants viewed 90 video clips depicting social and nonsocial actions in real-world contexts. Using arrangement tasks, we developed two behavioral models capturing judgments of action meaning (transitive purpose, social content), as well as three control models capturing judgments of visual content (people, objects, scene). We compared these models with semantic models based on word embeddings and visual models based on gaze and motion energy. Behavioral models of action meaning outperformed models of visual content, semantics, and low-level visual features across much of cortex. Transitivity and sociality captured large portions of unique variance throughout the action observation network, extending into regions not typically associated with action perception, such as ventral temporal cortex.

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30 min naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a logarithmic relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven’t kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a logarithmic relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

The 'default' mode of cognition involves simulating internal experiences. However, in some contexts, external focus can become captivating enough to act as the default mode. To examine how these orientations relate, we record fMRI activity in thirty-nine participants as they view naturalistic video clips, each one repeated four times in sequence. When instructed to direct attention externally on the video, most mind-wandering occurs on the last repetition and during less engaging clips. When instructed to direct attention internally on breathing, most mind-wandering occurs on the first repetition and for more engaging clips. Inter-subject correlation, within-subject correlation, and general linear models find similar fronto-parietal networks engaged during transitions from automatic to controlled attention across both conditions, indicating greater overlap in internal and external processing than previously assumed. We conclude that whether the automatic draw of attention is internal or external, and whether its disruptors are internal or external, is context-dependent.

Narrative comprehension is inherently context-sensitive, yet the brain and cognitive mechanisms by which brief contextual priming shapes story interpretation remain unclear. Using hidden Markov modeling (HMM) of fMRI data, we identified dynamic brain states as participants listened to an ambiguous spoken story under two distinct narrative contexts (affair vs. paranoia). We identified recurrent states involving auditory, language, and default mode network (DMN) regions that were expressed across both groups, as well as additional states characterized by recruitment of multiple-demand network (MDN) systems, including control, dorsal attention, and salience networks. Bayesian mixed-effects modeling revealed that contextual framing modulated how specific linguistic and character-related features influenced the probability of occupying these states. Complementary behavioral data showed parallel context-sensitive modulation of participants’ moment-to-moment interpretive judgments. Together, these findings suggest that contextual priming influences narrative comprehension through subtle, feature-dependent adjustments in the engagement of DMN- and MDN-related brain states during naturalistic story listening.

Large Language Models (LLMs) offer a framework for understanding language processing in the human brain. Unlike traditional models, LLMs represent words and context through layered numerical embeddings. Here, we demonstrate that LLMs’ layer hierarchy aligns with the temporal dynamics of language comprehension in the brain. Using electrocorticography (ECoG) data from participants listening to a 30-minute narrative, we show that deeper LLM layers correspond to later brain activity, particularly in Broca’s area and other language-related regions. We extract contextual embeddings from GPT-2 XL and Llama-2 and use linear models to predict neural responses across time. Our results reveal a strong correlation between model depth and the brain’s temporal receptive window during comprehension. We also compare LLM-based predictions with symbolic approaches, highlighting the advantages of deep learning models in capturing brain dynamics. We release our aligned neural and linguistic dataset as a public benchmark to test competing theories of language processing.

Recent research demonstrates that large language models can predict neural activity recorded via electrocorticography during natural language processing. To predict word-by-word neural activity, most prior work evaluates encoding models within individual electrodes and participants, limiting generalizability. Here we analyze electrocorticography data from eight participants listening to the same 30-min podcast. Using a shared response model, we estimate a common information space across participants. This shared space substantially enhances large language model-based encoding performance and enables denoising of individual brain responses by projecting back into participant-specific electrode spaces—yielding a 37% average improvement in encoding accuracy (from r = 0.188 to r = 0.257). The greatest gains occur in brain areas specialized for language comprehension, particularly the superior temporal gyrus and inferior frontal gyrus. Our findings highlight that estimating a shared space allows us to construct encoding models that better generalize across individuals.