Behaviorally prioritized entity-relation structure captures human visual cortical representations of natural scenes
Preprint
Abstract
Understanding natural scenes requires identifying visible entities and representing how those entities are related. Recent studies have shown that artificial neural networks (ANNs), large language models (LLMs), and vision language models (VLMs) can predict visual cortical responses to natural images. However, the neural organization of relational scene meaning remains poorly understood, in part because these models typically encode scene content in global feature spaces that are difficult to decompose into separable entity and relation components. Here, we combined scene-graph annotations, behavioral measurements, and large-scale neural datasets to characterize structured relational representations during natural vision. We used RotatE, a knowledge-graph embedding model, to represent head-relation-tail triplets annotated for images from the 7T Natural Scenes Dataset. Triplet embeddings reliably captured cortical representational structure across the visual hierarchy. Behavioral judgments further revealed systematic differences in triplet accessibility associated with visual, relational, and graph properties. Prioritizing more behaviorally accessible triplets improved neural correspondence and explained unique variance beyond object co-occurrence, ANN image features, and LLM caption embeddings. Decomposing triplet representations into entity and relation components revealed partially dissociable cortical contributions, with lateral parietal cortex showing sensitivity to both. Triplet-based semantic information also remained spatially grounded: visual-field-specific triplet models preferentially predicted voxels with matching retinotopic preferences. Finally, cross-species comparison indicated that triplet-based semantic features were relatively more aligned with human high-level visual cortex than with macaque inferotemporal cortex. Together, these findings provide new insights into the representation of semantic relational information in the human visual cortex during natural scene perception.Keywords
This publication has 46 references indexed in Scilit:
- Mapping concept and relational semantic representation in the brain using large language modelsCurrent Biology, 2026
- Scene Graph Generation: A comprehensive surveyNeurocomputing, 2023
- Feature-space selection with banded ridge regressionNeuroImage, 2022
- Panoptic Scene Graph Generation2022
- Typical visual-field locations facilitate access to awareness for everyday objectsCognition, 2018
- Exploring the Limits of Weakly Supervised Pretraining2018
- The feature-weighted receptive field: an interpretable encoding model for complex feature spacesNeuroImage, 2018
- Variable importance in regression modelsWIREs Computational Statistics, 2015
- Microsoft COCO: Common Objects in Context2014
- Modeling attention to salient proto-objectsNeural Networks, 2006