Thomas Lasman Botch

Language is a fundamental human capacity. Large language models (LLMs) have presented the first viable model of language outside of humans, yet how these models learn and use language differs significantly from humans. Here, we compare LLMs and humans predicting language with varying levels of sensory information—from disembodied written text to audiovisual videos of speakers—to demonstrate that, in both humans and LLMs, sensory context is critical for optimal performance. We asked human participants to predict upcoming words within narratives presented as either audiovisual, audio-only, or written language ( N = 1 , 500 total; 500 per modality), focusing on content words that convey essential meaning. We compared these predictions across modalities as well as to predictions generated by LLMs. Human predictions were overall more accurate than LLMs, regardless of the sensory modality of presentation. Compared to written language, both audiovisual and audio-only language increased the accuracy and consensus of human predictions and decreased alignment with LLMs. We identified that prosody, a signal that conveys important linguistic information via the auditory channel, partially drove both the observed advantage in accuracy within humans and divergence from LLMs. Integrating multimodal information—i.e., prosody, auditory, or audiovisual information—with the representation of language learned during LLM training improved models’ next-word prediction performance and increased the efficiency of language learning. These findings demonstrate that sensory contexts are foundational to human-like language behavior, and that these contexts can enrich and accelerate language acquisition within LLMs similar to what is observed within human development.

Large-scale brain activity reflects changes in attention. To understand how, we tested a hypothesis that the geometry of neural dynamics on the cortical attractor landscape, or movement along its “hills and valleys”, reflects attentional states. A dynamical systems model separating intrinsic dynamics from stimulus-driven influences was fit to fMRI data collected during rest, tasks, and movie-watching. Model simulations revealed a set of attractors aligned with canonical functional brain networks. The speed and direction of neural trajectories toward these attractors varied systematically with attentional states over time and across contexts. When participants were paying attention to effortful tasks, neural dynamics converged fast and directly toward a task-relevant attractor. In contrast, when participants were engaged in sitcom episodes, neural dynamics occupied a flatter region of the landscape, directed away from attractors. These findings demonstrate that while attractor locations are largely determined by cortical organization, the geometry of neural dynamics changes systematically across attentional states and contexts.

The brain is a complex dynamical system whose activity reflects changes in internal states, such as attention. While prior work has shown that large-scale brain activity reflects attention, the mechanism governing this association in a time-varying and task-dependent manner remains unknown. Here, we tested a hypothesis that the geometry of neural dynamics on the attractor landscape, or the movement along the "hills and valleys", reflects changes in attentional states over time and variations across controlled and naturalistic contexts. We fit a parametric dynamical systems model to fMRI data collected during rest, task performance, and naturalistic movie-watching. The model decomposes neural dynamics into components that are intrinsic versus extrinsically driven by stimuli. Model parameters were biologically meaningful, reflecting both cognitive states and individual differences. Model simulations revealed a set of attractors that mirrored functional brain networks, spanning the canonical gradient from sensorimotor to default mode network regions. The speed and direction of neural trajectories toward these attractors systematically varied across attentional states in a context-dependent manner. When participants were paying attention to effortful tasks, neural dynamics converged directly toward a task-relevant attractor, suggesting that it occupied a steeper region of the attractor landscape. In contrast, when participants were engaged in sitcom episodes, neural dynamics were in a flattened region of the landscape, directed away from the attractors. These findings demonstrate that while the positions of the attractors are largely determined by the cortical organization, the geometry of neural dynamics on the attractor landscape changes systematically across attentional states and situational contexts.

Different people listening to the same story may converge upon a largely shared interpretation while still developing idiosyncratic experiences atop that shared foundation. What linguistic properties support this individualized experience of natural language? Here, we investigate how the “concrete–abstract” axis—the extent to which a word is grounded in sensory experience—relates to within- and across-subject variability in the neural representations of language. Leveraging a dataset of human participants of both sexes who each listened to four auditory stories while undergoing functional magnetic resonance imaging, we demonstrate that neural representations of “concreteness” are both reliable across stories and relatively unique to individuals, while neural representations of “abstractness” are variable both within individuals and across the population. Using natural language processing tools, we show that concrete words exhibit similar neural representations despite spanning larger distances within a high-dimensional semantic space, which potentially reflects an underlying representational signature of sensory experience—namely, imageability—shared by concrete words but absent from abstract words. Our findings situate the concrete–abstract axis as a core dimension that supports both shared and individualized representations of natural language.

Different people listening to the same story may converge upon a largely shared interpretation while still developing idiosyncratic experiences atop that shared foundation. What semantic properties support this individualized experience of natural language? Here, we investigate how the “concreteness” of word meanings — i.e., the extent to which a concept is derived from sensory experience — relates to variability in the neural representations of language. Leveraging a large dataset of participants who each listened to four auditory stories while undergoing functional MRI, we demonstrate that an individual’s neural representations of concrete concepts are reliable across stories and unique to the individual. In contrast, we find that neural representations of abstract concepts are variable both within individuals and across the population. Using natural language processing tools, we show that concrete words exhibit similar neural signatures despite spanning larger distances within a high-dimensional semantic space, which potentially reflects an underlying signature of sensory experience — namely, imageability — shared by concrete words but absent from abstract words. Our findings situate the concrete-abstract semantic axis as a core dimension that supports reliable yet individualized representations of natural language.

Each view of our environment captures only a subset of our immersive surroundings. Yet, our visual experience feels seamless. A puzzle for human neuroscience is to determine what cognitive mechanisms enable us to overcome our limited field of view, and efficiently anticipate new views as we sample our visual surroundings. Here, we tested whether memory-based predictions ofupcoming scene views facilitate efficient perceptual judgments across head turns. We tested this hypothesis using immersive, head-mounted virtual reality (VR). After learning a set of 360° real-world environments, participants (N=64 across 3 experiments) were briefly primed with a single view from a studied environment, and then turned left or right to make a perceptual judgment about an adjacent scene view. We found that participants’ perceptual judgments were faster when they were primed with images from the same (vs. neutral or different) environments. Importantly, priming required memory: it only occurred in learned (vs. novel) environments, where the link between adjacent scene views was known. Further, priming only occurred in the direction of planned head turns, consistent with a role in supporting active vision. Taken together, we propose that memory-based predictions facilitate rapid perception across large-scale visual actions, such as head and body movements, and may be critical for efficient behavior in complex immersive environments.

Visual search is a ubiquitous activity in real-world environments. Yet, traditionally, visual search is investigated in tightly controlled paradigms, where head-restricted participants locate a minimalistic target in a cluttered array that is presented on a computer screen. Do traditional visual search tasks predict performance in naturalistic settings, where participants actively explore complex, real-world scenes? Here, we leverage advances in virtual reality technology to test the degree to which classic and naturalistic search are limited by a common factor, set size, and the degree to which individual differences in classic search behavior predict naturalistic search behavior in a large sample of individuals (N = 75). In a naturalistic search task, participants looked for an object within their environment via a combination of head-turns and eye-movements using a head-mounted display. Then, in a classic search task, participants searched for a target within a simple array of colored letters using only eye-movements. In each task, we found that participants’ search performance was impacted by increases in set size—the number of items in the visual display. Critically, we observed that participants’ efficiency in classic search tasks—the degree to which set size slowed performance—indeed predicted efficiency in real-world scenes. These results demonstrate that classic, computer-based visual search tasks are excellent models of active, real-world search behavior.

Individuals with autism spectrum conditions (ASC) describe differences in both social cognition and sensory processing, but little is known about the causal relationship between these disparate functional domains. In the present study, we sought to understand how a core characteristic of autism—reduced social attention—is impacted by the complex multisensory signals present in real‐world environments. We tested the hypothesis that reductions in social attention associated with autism would be magnified by increasing perceptual load (e.g., motion, multisensory cues). Adult participants (N = 40; 19 ASC) explored a diverse set of 360° real‐world scenes in a naturalistic, active viewing paradigm (immersive virtual reality + eyetracking). Across three conditions, we systematically varied perceptual load while holding the social and semantic information present in each scene constant. We demonstrate that reduced social attention is not a static signature of the autistic phenotype. Rather, group differences in social attention emerged with increasing perceptual load in naturalistic environments, and the susceptibility of social attention to perceptual load predicted continuous measures of autistic traits across groups. Crucially, this pattern was specific to the social domain: we did not observe differential impacts of perceptual load on attention directed toward nonsocial semantic (i.e., object, place) information or low‐level fixation behavior (i.e., overall fixation frequency or duration). This study provides a direct link between social and sensory processing in autism. Moreover, reduced social attention may be an inaccurate characterization of autism. Instead, our results suggest that social attention in autism is better explained by “social vulnerability,” particularly to the perceptual load of real‐world environments.

Differences in both social and sensory processing are well described in the autism literature, but little is known about the causal relationship between these domains, particularly in real-world environments. In the present study, we sought to understand how a core characteristic of the autistic phenotype – reduced social attention – is impacted by the complex multisensory signals present in environments like those encountered in daily life. We tested the hypothesis that autistic group differences in social attention would be magnified by increasing perceptual load (e.g., motion, multisensory cues). Adult participants (N = 40; 19 ASC) explored a diverse set of 360° real-world scenes in a naturalistic, active viewing paradigm (immersive virtual reality + eyetracking). Across three conditions, we systematically varied perceptual load while holding the social and semantic information present in each scene constant. We demonstrate that group differences in social attention are not a static signature of the autistic phenotype; rather, they emerge with increasing perceptual load in naturalistic environments. Crucially, this pattern was specific to the social domain and not driven by low-level oculomotor differences: we did not observe differential impacts of perceptual load on attention directed toward nonsocial semantic (i.e., object, place) information or fixation behavior. This study provides a direct link between social and sensory processing in autism. Moreover, reduced social attention may be an inaccurate characterization of autism. Instead, our results suggest that social attention in autism is better explained by “social vulnerability", particularly to the perceptual load of real-world environments.