Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Authors:
- Rakshith ShettyRSRakshith Shetty
- Marcus RohrbachMRMarcus Rohrbach
- Lisa Anne HendricksLHLisa Anne Hendricks
- Mario FritzMFMario Fritz
- Bernt SchieleBSBernt Schiele
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4155-4164
Abstract
While strong progress has been made in image captioning recently, machine and human captions are still quite distinct. This is primarily due to the deficiencies in the generated word distribution, vocabulary size, and strong bias in the generators towards frequent captions. Furthermore, humans - rightfully so - generate multiple, diverse captions, due to the inherent ambiguity in the captioning task which is not explicitly considered in today's systems. To address these challenges, we change the training objective of the caption generator from reproducing ground-truth captions to generating a set of captions that is indistinguishable from human written captions. Instead of handcrafting such a learning target, we employ adversarial training in combination with an approximate Gumbel sampler to implicitly match the generated distribution to the human one. While our method achieves comparable performance to the state-of-the-art in terms of the correctness of the captions, we generate a set of diverse captions that are significantly less biased and better match the global uni-, bi- and tri-gram distributions of the human captions.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Exploiting Scene Context for Image Captioning2016
- Deep Residual Learning for Image Recognition2016
- Image Captioning with Semantic Attention2016
- Reasoning about Pragmatics with Neural Listeners and Speakers2016
- Long-term recurrent convolutional networks for visual recognition and description2015
- CIDEr: Consensus-based image description evaluation2015
- Language Models for Image Captioning: The Quirks and What Works2015
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Translating Video Content to Natural Language Descriptions2013
- Every Picture Tells a Story: Generating Sentences from Images2010