Context-aware Captions from Context-agnostic Supervision
Preprint
Abstract
We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of "siamese cat" and "tiger cat", we generate language that describes the "siamese cat" in a way that distinguishes it from "tiger cat". Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Generating Visual Explanations2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- Show and tell: A neural image caption generator2015
- From captions to visual concepts and back2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Microsoft COCO: Common Objects in Context2014
- Bringing Semantics into Focus Using Visual Abstraction2013
- Image description with a goal: Building efficient discriminating expressions for images2012