Nancy Kanwisher

How does the brain understand the physical world, predict future events, and plan actions? Are these problems solved by a unified world model, as hypothesized by cognitive scientists and artificial intelligence researchers? Posterior parietal cortex has been implicated in all these functions, but whether the same neurons support each of them is unknown. We recorded from the superior parietal lobule (SPL) of two tetraplegic participants implanted with microelectrode arrays. SPL units preferentially engaged by intuitive physics, judging block-tower stability and viewing naturalistic physical events, also predicted whether moving objects would collide hundreds of milliseconds before contact, generalizing between different physical scenarios. These populations also encoded which hand action the participant intended to perform. These findings identify SPL as a substrate for physical world modeling.

How do neurons code information? Recent work emphasizes the geometry and information content of population codes, using measures that are blind to the native tunings (or ‘axes’) of neural responses. But might these representational axes matter, with some privileged systematically over others? We developed methods to test for alignment of neural tuning across brains and deep convolutional neural networks (DCNNs). Across both vision and audition, both brains and DCNNs consistently favoured certain axes for representing the natural world. Moreover, the representational axes of DCNNs trained on natural inputs were aligned to those in perceptual cortices, such that axis-sensitive model–brain similarity metrics better differentiated competing models of biological sensory systems. We further show that coding schemes that privilege certain axes can facilitate task performance, reduce downstream wiring costs and improve generalization. These results motivate a new framework for understanding neural tuning in biological and artificial networks and its computational benefits.

Artificial neural networks (ANNs) can predict neural responses to natural images with remarkable accuracy, making them promising models of human vision. However, prediction scores alone reveal little about what computations these models have captured or how well they generalize beyond the images on which they are trained. Here, we introduce a principled zero-shot model evaluation framework in which ANN-based encoding models are frozen before being tested on not just held out images but entirely new datasets. Our strongest tests repurpose decades of cognitive neuroscience experiments as diagnostic benchmarks for identifying which computations current models capture and where they fail. To implement this framework we built encoding models of human category-selective regions (FFA, PPA, and EBA) using several pretrained ANNs and evaluated these fixed models across 7 independent fMRI datasets and 31 cognitive experiments from 10 published studies. This framework revealed three broad patterns. First, zero-shot evaluations exposed systematic limits to model predictivity that were largely hidden by standard within-dataset cross-validation. These limits were structured varying across brain regions and model classes and depending strongly on the similarity between the images used to build and test the models. Second, current models reproduced some, but not all, classical findings from cognitive neuroscience, providing a diagnosis of the computations that current models have yet to capture. Finally, models that performed well on prediction tests also tended to succeed at reproducing cognitive neuroscience findings, suggesting that prediction and explanation are closely linked. Unlike prediction scores however, cognitive tests help diagnose where and why models fail. Together, the zero-shot evaluation framework provides a scalable and unified approach for evaluating brain models across prediction and cognitive neuroscience tests, which can help us better understand what current models capture and what still remains to be explained.

Sighted people rely on vision to recognize and navigate the local environment. By adulthood, human cortex contains at least three regions that respond selectively to visual scene information, but it remains unknown when or how these regions develop. One hypothesis is that development of scene selectivity depends on passive exposure to the low-level visual statistics of scenes. Another hypothesis is that development depends on active experience using scene information to plan and guide navigation. Using ecological momentary assessment, we measured the quantity of forward-facing ego-motion and active navigation that infants experienced in the first months of life. By age 5 mo, infants cumulatively experienced around 340 h of forward facing ego-motion, almost exclusively passive. Next we used functional MRI in infants aged 2 to 9 mo to measure neural responses to scenes. Awake infants watched videos of ego-motion through real-world scenes, as well as videos of faces, objects, and scrambled videos. We found stronger responses to scenes than control conditions in the location of all three cortical scene regions. Responses in infant scene regions a) were dissociable from those in nearby face and object regions; b) could not be fully explained by low-level visual features; and c) were found even in subsamples of infants with no locomotor experience. Thus, the basic signature of scene selectivity emerges prior to independent navigation and with limited passive visual exposure to the diagnostic visual statistics of scenes.

The functional organization of human cortex has been mapped in considerable detail ^1–5 , yet the axonal connections linking these areas remain largely unknown. This gap precludes inferring either the computational pathways through cortical networks or the principles governing why each area connects to its particular target regions. Here, in patients undergoing intracranial monitoring for epilepsy, we used concurrent electrical stimulation and functional magnetic resonance imaging (es-fMRI), validated previously against tracer studies in nonhuman primates ⁶ , to map anatomical connectivity of cortical sites across the whole brain, and combined these maps with task and resting-state fMRI in the same individuals. Es-fMRI revealed four main findings. First, connectivity followed an asymmetric functional-similarity principle: connected sites tended to share similar functional profiles, but, because connectivity is sparse, most pairs of functionally similar sites were not connected. Second, although connectivity also declines with distance ⁷ , the functional profile explained three times as much variance in connectivity as distance did, and long-range connections were the most functionally specific. Third, es-fMRI revealed direct monosynaptic links between established functional regions, including between the fusiform face area (FFA) ² and the temporoparietal junction (TPJ) ³ for social cognition. Fourth, resting-state functional connectivity (rsFC) of a stimulation site was only weakly correlated with that site’s es-fMRI connectivity. Together, these results provide a first principled link between anatomical connectivity and functional organization in the human cortex, and establish es-fMRI as a scalable approach to building a tracer-grade connectome of the human brain.

Language understanding entails not just extracting the surface-level meaning of the linguistic input, but constructing rich mental models of the situation it describes. Here we propose that because processing within the brain's core language system is fundamentally limited, deeply understanding language requires exporting information from the language system to other brain regions that compute perceptual and motor representations, construct mental models, and store our world knowledge and autobiographical memories. We review the existing evidence for this hypothesis, and argue that recent progress in cognitive neuroscience provides both the conceptual foundation and the methods to directly test it, thus opening up a new strategy to reveal what it means, cognitively and neurally, to understand language.

One of the most robust findings in human cognitive neuroscience is the discovery that many regions of the cortex are engaged in distinctive, often very specific, functions. Although these regions are found in approximately the same location in almost all typical participants, their exact location varies from one individual to the next. Thus, the first step in studying these regions is to identify them in each participant individually. Standard functional localizers have been devised to accomplish this goal, but most localizers identify only a few regions. Many important questions in modern neuroscience can only be answered by measuring the responses of multiple cortical regions at the same time. Here, we introduce a new Efficient Multifunction fMRI localizer (EMFL) in which visual and auditory conditions are presented simultaneously, enabling the identification, in just 14 minutes of fMRI scan time, of 14 of the most widely-studied cortical regions: those selectively engaged in perceiving faces, places, bodies, words, objects, and speech sounds; understanding language and other people’s thoughts; and engaging broadly in demanding cognitive tasks (the “multiple demand” system). We validate the EMFL by showing that it identifies the major functional regions of interest as well as the standard localizers do, in a quarter of the scan time. The stimuli and presentation code for this new localizer are publicly available, enabling future studies to efficiently identify the major functional regions of interest with the same procedure across labs.

The ventral, dorsal, and lateral streams in high-level human visual cortex are implicated in distinct functional processes. Yet, deep neural networks (DNNs) trained on a single task model the entire visual system surprisingly well, hinting at common computational principles across these pathways. To explore this inconsistency, we applied a novel sparse decomposition approach to identify the dominant components of visual representations within each stream. Consistent with traditional neuroscience research, we find a clear difference in component response profiles across the three visual streams -- identifying components selective for faces, places, bodies, text, and food in the ventral stream; social interactions, implied motion, and hand actions in the lateral stream; and some less interpretable components in the dorsal stream. Building on this, we introduce Sparse Component Alignment (SCA), a new method for measuring representational alignment between brains and machines that better captures the latent neural tuning of these two visual systems. Using SCA, we find that standard visual DNNs are more aligned with the ventral than either dorsal or lateral representations. SCA reveals these distinctions with greater resolution than conventional population-level geometry, offering a measure of representational alignment that is sensitive to a system's underlying axes of neural tuning.

Sighted people rely on vision to recognize and navigate the local environment. By adulthood, human cortex contains at least three regions that respond selectively to visual scene information, but it remains unknown when or how these regions develop. One hypothesis is that scene selectivity emerges gradually in regions that initially prefer certain low-level visual features (e.g., peripheral visual input, high spatial frequencies, rectilinearity), and then exposure to the visual statistics of natural scenes drives the emergence of scene selective responses. However, both aspects of this hypothesis remain to be tested: how early scene selectivity first arises in human development, and whether it is driven by passive exposure to visual statistics. We therefore collected functional magnetic resonance imaging data from awake 2–9-month-old infants while they watched videos of real-world scenes with ego-motion, as well as faces, objects, and scrambled videos. We found stronger responses to scenes than control conditions in the location of all three scene regions. Scene-selective responses could not be explained by low-level visual properties of the stimuli, and were found in infants as young as 2-5 months old, with no evidence of age-related change. We also measured infants’ experience independently navigating (e.g., crawling), which was not necessary for the development of scene-selectivity. In sum, cortical regions are scene-selective in human infants prior to independent navigation, and after only limited exposure to visual scene statistics. Significance statement: Despite extensive work on the functional organization of scene processing in the human adult visual cortex, little is known about the developmental origins of category selectivity for visual scenes. Here we used fMRI in awake human infants to discover that all three regions of the known visual scene processing system are present within the first few months of life – the youngest age yet detected. Scene-selective cortex therefore develops after only a few months of limited visual exposure, and prior to active experience using visual scene information to plan and guide independent navigation (e.g., by crawling). These findings provide a fundamental constraint on theories of how scene selectivity develops in high-level visual cortex.

Successful engagement with the physical world requires the ability to predict future events and plan interventions to alter that future. Growing evidence implicates a set of regions in the human parietal and frontal lobes [also known as the “physics network” (PN)] in such intuitive physical inferences. However, the central tenet of this hypothesis, that PN runs forward simulations to predict future states, remains untested. In this preregistered study, we first show that PN abstractly represents whether two objects are in contact with each other, a physical scene property critical for prediction (because objects’ fates are intertwined when they are in contact). We then show that PN (but not other visual areas) carries abstract information about predicted future contact events (i.e., collisions). These findings support the hypothesis that PN contains a generative model of the physical world that conducts forward simulations, serving as the brain’s “physics engine.”