Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10687-10696
Abstract
Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions. To address this concern, some variational auto-encoder (VAE) and generative adversarial net (GAN) based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since our summary should drive caption generation. We achieve the trifecta: (1) High accuracy for the diverse captions as evaluated by standard captioning metrics and user studies; (2) Faster computation of diverse captions compared to beam search and diverse beam search; and (3) High diversity as evaluated by counting novel sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Towards Diverse and Natural Image Descriptions via a Conditional GAN2017
- Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training2017
- Boosting Image Captioning with Attributes2017
- Self-Critical Sequence Training for Image Captioning2017
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning ChallengeIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Situation Recognition: Visual Semantic Role Labeling for Image Understanding2016
- Deep visual-semantic alignments for generating image descriptions2015
- CIDEr: Consensus-based image description evaluation2015
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Solving the problem of cascading errors2006