Self-Guiding Multimodal LSTM—When We Do Not Have a Perfect Training Dataset for Image Captioning
- 22 May 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (11) , 5241-5252
Abstract
In this paper, a self-guiding multimodal LSTM (sgLSTM) image captioning model is proposed to handle uncontrolled imbalanced real-world image-sentence dataset. We collect FlickrNYC dataset from Flickr as our testbed with 306,165 images and the original text descriptions uploaded by the users are utilized as the ground truth for training. Descriptions in FlickrNYC dataset vary dramatically ranging from short term-descriptions to long paragraph-descriptions and can describe any visual aspects, or even refer to objects that are not depicted. To deal with the imbalanced and noisy situation and to fully explore the dataset itself, we propose a novel guiding textual feature extracted utilizing a multimodal LSTM (mLSTM) model. Training of mLSTM is based on the portion of data in which the image content and the corresponding descriptions are strongly bonded. Afterwards, during the training of sgLSTM on the rest training data, this guiding information serves as additional input to the network along with the image representations and the ground-truth descriptions. By integrating these input components into a multimodal block, we aim to form a training scheme with the textual information tightly coupled with the image content. The experimental results demonstrate that the proposed sgLSTM model outperforms the traditional state-of-the-art multimodal RNN captioning framework in successfully describing the key components of the input images.Keywords
All Related Versions
Funding Information
- National Science Foundation (EFRI-1137172, IIS-1400802)
This publication has 33 references indexed in Scilit:
- Deep visual-semantic alignments for generating image descriptions2015
- From captions to visual concepts and back2015
- CIDEr: Consensus-based image description evaluation2015
- Language Models for Image Captioning: The Quirks and What Works2015
- Déjà Image-Captions: A Corpus of Expressive Descriptions in Repetition2015
- A Distributed Representation Based Query Expansion Approach for Image Captioning2015
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- Glove: Global Vectors for Word Representation2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Long Short-Term MemoryNeural Computation, 1997