Human-like meaning maps from single-prompt VLM ratings of local scene meaning
Preprint
Abstract
Visual attention is shaped both by low-level perceptual features and by the semantic properties of scene regions. However, compared to the salience of low-level image features, the characterization of image properties that lead to scene understanding has been more difficult. The semantic properties of scene regions have been operationalized via meaning maps, proposed by Henderson and Hayes (2017), by having human observers rate the informativeness of scene patches. However, this rating procedure is labor-intensive and limits the method’s versatility. An automated alternative, the DeepMeaning model, reduces rating costs but still requires human training data from a similar class of images, limiting generalizability. Here, we tested whether instruction-following vision-language models (VLMs) can generate human-like ratings in a zero-shot manner: using the original human-rating instructions as a single prompt without any task-specific training. We obtained meaningfulness ratings from both open-weight and proprietary VLMs for scene patches across three meaning-map datasets, and compared the resulting zero-shot AI meaning maps (AIMMs) to human meaning maps and maps obtained from the DeepMeaning model. Zero-shot AIMMs closely approximate human meaning maps, and maps from the task-specific DeepMeaning model, showing that VLMs can replicate human semantic judgments without any training. Zero-shot AIMMs demonstrate utility and validity, matching human meaning maps in their ability to predict human fixations and reproducing the same pattern of relative performance against other gaze-prediction models. We provide open-source code that allows researchers to use this zero-shot approach to quickly and flexibly explore meaning maps across spatial scales, prompts, and scene categories.Keywords
This publication has 51 references indexed in Scilit:
- Finding the meaning in meaning maps: Quantifying the roles of semantic and non-semantic scene information in guiding visual attentionCognition, 2024
- Knowledge-driven perceptual organization reshapes information sampling via eye movements.Journal of Experimental Psychology: Human Perception and Performance, 2023
- Meaning maps capture the density of local semantic features in scenes: A reply to Pedziwiatr, Kümmerer, Wallis, Bethge & Teufel (2021)Cognition, 2021
- Are emotional objects visually salient? The Emotional Maps DatabaseJournal of Visual Communication and Image Representation, 2021
- Meaning maps and saliency models based on deep convolutional neural networks are insensitive to image meaning when predicting human fixationsCognition, 2021
- Integrative Benchmarking to Advance Neurally Mechanistic Models of Human IntelligenceNeuron, 2020
- Where the action could be: Speakers look at graspable objects and meaningful scene regions when describing potential actions.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2020
- Saliency from hierarchical adaptation through decorrelation and variance normalizationImage and Vision Computing, 2012
- Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.Psychological Review, 2006
- Eye Movements and Vision1967