High-level visual representations in the human brain are aligned with large language models
Top Cited Papers
Open Access
- 7 August 2025
- journal article
- research article
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 7 (8) , 1220-1234
Abstract
The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of scene captions successfully characterize brain activity evoked by viewing the natural scenes. This mapping captures selectivities of different brain areas and is sufficiently robust that accurate scene captions can be reconstructed from brain activity. Using carefully controlled model comparisons, we then proceed to show that the accuracy with which LLM representations match brain representations derives from the ability of LLMs to integrate complex information contained in scene captions beyond that conveyed by individual words. Finally, we train deep neural network models to transform image inputs into LLM representations. Remarkably, these networks learn representations that are better aligned with brain representations than a large number of state-of-the-art alternative models, despite being trained on orders-of-magnitude less data. Overall, our results suggest that LLM embeddings of scene captions provide a representational format that accounts for complex information extracted by the brain from visual inputs.Keywords
This publication has 110 references indexed in Scilit:
- VLP: A Survey on Vision-language Pre-trainingMachine Intelligence Research, 2023
- Color-biased regions in the ventral visual pathway are food selectiveCurrent Biology, 2022
- Reading scenes: how scene grammar guides attention and aids perception in real-world environmentsCurrent Opinion in Psychology, 2019
- The spatiotemporal neural dynamics underlying perceived similarity for real-world objectsNeuroImage, 2019
- Exploring the Limits of Weakly Supervised Pretraining2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Microsoft COCO: Common Objects in Context2014
- Matching Categorical Object Representations in Inferior Temporal Cortex of Man and MonkeyNeuron, 2008
- Predicting the Stream of Consciousness from Activity in Human Visual CortexCurrent Biology, 2005
- Separate visual pathways for perception and actionTrends in Neurosciences, 1992