Semantic Image Retrieval via Active Grounding of Visual Situations
Authors:
- Max H. QuinnMQMax H. Quinn
- Erik ConserECErik Conser
- Jordan M. WitteJWJordan M. Witte
- Melanie MitchellMMMelanie Mitchell
- 1 January 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 172-179
Abstract
We describe a novel architecture for semantic image retrieval-in particular, retrieval of instances of visual situations. Visual situations are concepts such as “a boxing match,” “walking the dog,” “a crowd waiting for a bus,” or “a game of pingpong,” whose instantiations in images are linked more by their common spatial and semantic structure than by low-level visual similarity. Given a query situation description, our architecture-called Situate-learns models capturing the visual features of expected objects as well the expected spatial configuration of relationships among objects. Given a new image, Situate uses these models in an attempt to ground (i.e., to create a bounding box locating) each expected component of the situation in the image via an active search procedure. Situate uses the resulting grounding to compute a score indicating the degree to which the new image is judged to contain an instance of the situation. Such scores can be used to rank images in a collection as part of a retrieval system. In the preliminary study described here, we demonstrate the promise of this system by comparing Situate's performance with that of two baseline methods, as well as with a related semantic image-retrieval system based on “scene graphs.”Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Object-Scene Convolutional Neural Networks for event recognition in images2015
- Image retrieval using scene graphs2015
- An active search strategy for efficient object class detection2015
- A survey on still image based human action recognitionPattern Recognition, 2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Expectation (and attention) in visual cognitionTrends in Cognitive Sciences, 2009
- What, where and who? Classifying events by scene and object recognition2007
- Scene context guides eye movements during visual searchVision Research, 2006
- Animate visionArtificial Intelligence, 1991