Predicting upcoming visual features during eye movements yields scene representations aligned with human visual cortex
Open Access
- 23 October 2026
- journal article
- research article
- Published by American Association for the Advancement of Science (AAAS) in Science Advances
- Vol. 12 (43) , eaee1955
Abstract
Natural scenes are complex arrangements of objects, surfaces, and backgrounds. For the brain’s visual system to effectively operate, it needs to extract not only what objects are present but also their spatial and semantic relations. We hypothesize that such structures may be learned, in a self-supervised fashion, by exploiting temporal regularities of natural active vision: Each fixation reveals a glimpse that is related to the previous one via co-occurrence and saccade-conditioned spatial regularities. We instantiate this idea with glimpse prediction networks (GPNs), recurrent models trained to predict the embedding of the next glimpse along human-like scanpaths. GPNs are shown to successfully extract complex scene information, including object co-occurrences and spatial object arrangements, and integrate information across glimpses. GPN representations align strongly with human fMRI responses in mid- and higher-level visual cortex and match, often outperform, alternative state-of-the-art ANN models, establishing next-glimpse-prediction as a biologically plausible route toward brain-aligned scene representations.Keywords
Funding Information
- European Research Council (101039524)
- Schweizerischer Nationalfonds (203018)
This publication has 46 references indexed in Scilit:
- Zero-shot counting with a dual-stream neural network modelNeuron, 2024
- Bridging the data gap between children and large language modelsTrends in Cognitive Sciences, 2023
- MaxViT: Multi-axis Vision Transformer2022
- Structure learning and the posterior parietal cortexProgress in Neurobiology, 2020
- Object Vision in a Structured WorldTrends in Cognitive Sciences, 2019
- Transformation from independent to integrative coding of multi-object arrangements in human visual cortexNeuroImage, 2018
- Category selectivity in human visual cortex: Beyond visual object recognitionNeuropsychologia, 2017
- Microsoft COCO: Common Objects in Context2014
- Tensor product variable binding and the representation of symbolic structures in connectionist systemsArtificial Intelligence, 1990
- Eye Movements During Perception of Complex Objects1967