Referring Relationships
Authors:
- Ranjay KrishnaRKRanjay Krishna
- Ines ChamiICInes Chami
- Michael BernsteinMBMichael Bernstein
- Li Fei-FeiLFLi Fei-Fei
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6867-6876
Abstract
Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets - CLEVR, VRD and Visual Genome - but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Conditional Random Fields as Recurrent Neural Networks2015
- Image retrieval using scene graphs2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval2015
- COSTA: Co-Occurrence Statistics for Zero-Shot Classification2014
- Learning to share visual appearance for multiclass object detection2011
- Recognition using visual phrases2011
- Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.Psychological Review, 2006
- Shifting attention in visual space: Tests of moving-spotlight models versus an activity-distribution model.Journal of Experimental Psychology: Human Perception and Performance, 1997
- Episodic theory of the dynamics of spatial attention.Psychological Review, 1995