Weakly-Supervised Visual Grounding of Phrases with Linguistic Structures
Authors:
- Fanyi XiaoFXFanyi Xiao
- Leonid SigalLSLeonid Sigal
- Yong Jae LeeYLYong Jae Lee
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 5253-5262
Abstract
We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i.e., localize) arbitrary linguistic phrases, in the form of spatial attention masks. Specifically, the model is trained with images and their associated image-level captions, without any explicit region-to-phrase correspondence annotations. To this end, we introduce an end-to-end model which learns visual groundings of phrases with two types of carefully designed loss functions. In addition to the standard discriminative loss, which enforces that attended image regions and phrases are consistently encoded, we propose a novel structural loss which makes use of the parse tree structures induced by the sentences. In particular, we ensure complementarity among the attention masks that correspond to sibling noun phrases, and compositionality of attention masks among the children and parent phrases, as defined by the sentence parse tree. We validate the effectiveness of our approach on the Microsoft COCO and Visual Genome datasets.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Long-term recurrent convolutional networks for visual recognition and description2015
- Mind's eye: A recurrent visual representation for image caption generation2015
- From image-level to pixel-level labeling with Convolutional Networks2015
- Deep visual-semantic alignments for generating image descriptions2015
- Fully convolutional networks for semantic segmentation2015
- Multi-fold MIL Training for Weakly Supervised Object Localization2014
- Tell Me What You See and I Will Show You Where It Is2014
- Object class recognition by unsupervised scale-invariant learning2003
- Unsupervised Learning of Models for Recognition2000
- Long Short-Term MemoryNeural Computation, 1997