Context-Aware Captions from Context-Agnostic Supervision
Authors:
- Ramakrishna VedantamRVRamakrishna Vedantam
- Samy BengioSBSamy Bengio
- Kevin MurphyKMKevin Murphy
- Devi ParikhDPDevi Parikh
- Gal ChechikGCGal Chechik
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 1070-1079
Abstract
We introduce an inference technique to produce discriminative context-aware image captions (captions that describe differences between images or visual concepts) using only generic context-agnostic training data (captions that describe a concept or an image in isolation). For example, given images and captions of siamese cat and tiger cat, we generate language that describes the siamese cat in a way that distinguishes it from tiger cat. Our key novelty is that we show how to do joint inference over a language model that is context-agnostic and a listener which distinguishes closely-related concepts. We first apply our technique to a justification task, namely to describe why an image contains a particular fine-grained category as opposed to another closely-related category of the CUB-200-2011 dataset. We then study discriminative image captioning to generate language that uniquely refers to one of two semantically-similar images in the COCO dataset. Evaluations with discriminative ground truth for justification and human studies for discriminative image captioning reveal that our approach outperforms baseline generative and speaker-listener approaches for discrimination.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Visual Madlibs: Fill in the Blank Description Generation and Question Answering2015
- VQA: Visual Question Answering2015
- Deep visual-semantic alignments for generating image descriptions2015
- CIDEr: Consensus-based image description evaluation2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Asking for Help Using Inverse Semantics2014
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- Not Everybody's Special: Using Neighbors in Referring Expressions with Uncertain Attributes2013
- Simultaneous Active Learning of Classifiers & Attributes via Relative Feedback2013
- Long Short-Term MemoryNeural Computation, 1997