During natural vision, semantic novelty modulates fixation-related processing in primate cortex

Authors:
Preprint
Publisher website
Google Scholar
Abstract
We sample visual scenes with short gaze fixations separated by saccades. While low-level integration is known, semantic integration of foveal vision across multiple fixations remains unclear. We hypothesized that the brain responds to changes in semantic information from one fixation to the next, and therefore postulated a neural signal associated with semantic novelty for each saccade. Novelty was measured using a deep network on foveal vision. Novelty modulated frontal and occipital fixation-related potentials in human EEG during natural viewing of full-length movies (3.4×10 6 saccades). Intracranial recordings in humans (9.0×10 4 saccades) and non-human primates (3.3×10 4 saccades) revealed broadband high-frequency activity modulations in ventromedial visual and frontal brain areas. This modulation was stronger for movies than static images, and frontal modulation preceded occipital modulation, suggesting top-down effects. This modulation of fixation-related activity with novelty suggests that foveal representations are integrated across saccades to construct scene representations during natural viewing in primates. Significance statement: Primate vision relies on eye movements to sample the world, yet how the brain integrates high-level meaning across these “snapshots” remains a mystery. By combining deep-learning models with massive electrophysiological datasets from humans and macaques, we show that the brain responds to changes in semantic content between fixations. We identified a “semantic novelty” signal that modulates neural activity across the primate cortex, appearing in frontal areas before a saccade is completed. These findings suggest visual processing is not a series of independent glimpses, but a continuous process integrating foveal information at the semantic level. This work bridges biological vision and artificial intelligence, providing a new framework for understanding how the brain maintains a coherent understanding of a dynamic world.

This publication has 108 references indexed in Scilit: