Guiding the Long-Short Term Memory Model for Image Caption Generation
Authors:
- Xu JiaXJXu Jia
- Efstratios GavvesEGEfstratios Gavves
- Basura FernandoBFBasura Fernando
- Tinne TuytelaarsTTTinne Tuytelaars
Top Cited Papers
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2407-2415
Abstract
In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- CIDEr: Consensus-based image description evaluation2015
- TreeTalk: Composition and Compression of Trees for Image DescriptionsTransactions of the Association for Computational Linguistics, 2014
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- BabyTalk: Understanding and Generating Simple Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Meteor2007
- Long Short-Term MemoryNeural Computation, 1997
- RELATIONS BETWEEN TWO SETS OF VARIATESBiometrika, 1936