Comprehension-Guided Referring Expressions
Authors:
- Ruotian LuoRLRuotian Luo
- Gregory ShakhnarovichGSGregory Shakhnarovich
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 3125-3134
Abstract
We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic image captioning which lacks natural standard evaluation criteria, quality of a referring expression may be measured by the receivers ability to correctly infer which object is being described. Following this intuition, we propose two approaches to utilize models trained for comprehension task to generate better expressions. First, we use a comprehension module trained on human-generated expressions, as a critic of referring expression generator. The comprehension module serves as a differentiable proxy of human evaluation, providing training signal to the generation module. Second, we use the comprehension model in a generate-and-rerank pipeline, which chooses from candidate expressions generated by a model according to their performance on the comprehension task. We show that both approaches lead to improved referring expression generation on multiple benchmark datasets.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Learning Deep Structure-Preserving Image-Text Embeddings2016
- Learning Deep Representations of Fine-Grained Visual Descriptions2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Fast R-CNN2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- Speech recognition with deep recurrent neural networks2013
- The segmented and annotated IAPR TC-12 benchmarkComputer Vision and Image Understanding, 2010
- Long Short-Term MemoryNeural Computation, 1997