Improving Diversity of Image Captioning Through Variational Autoencoders and Adversarial Learning
- 1 January 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 263-272
Abstract
Learning translation from images to human-readable natural language has become a great challenge in computer vision research in recent years. Existing works explore the semantic correlation between the visual and language domains via encoder-to-decoder learning frameworks based on classifying visual features in the language domain. This approach, however, is criticized for its lacking of naturalness and diversity. In this paper, we demonstrate a novel way to learn a semantic connection between visual information and natural language directly based on a Variational Autoencoder (VAE) that is trained in an adversarial routine. Instead of using the classification based discriminator, our method directly learns to estimate the diversity between a hidden vector embedded from a text encoder and an informative feature that is sampled from a learned distribution of the autoencoders. We show that the sentences learned from this matching contains accurate semantic meaning with high diversity in the image captioning task. Our experiments on the popular MSCOCO dataset indicates that our method learns to generate high-quality natural language with competitive scores on both correctness and diversity.Keywords
This publication has 31 references indexed in Scilit:
- Factorized Similarity Learning in Networks2014
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Convolutional Neural Networks for Sentence Classification2014
- Equivalence of distance-based and RKHS-based statistics in hypothesis testingThe Annals of Statistics, 2013
- Recommending Flickr groups with social topic modelInformation Retrieval Journal, 2012
- On clustering heterogeneous social media objects with outlier links2012
- Refining video annotation by exploiting pairwise concurrent relation2007
- Typicality ranking via semi-supervised multiple-instance learning2007
- Video annotation based on temporally consistent Gaussian random fieldElectronics Letters, 2007
- Long Short-Term MemoryNeural Computation, 1997