Deep Hierarchical Encoder–Decoder Network for Image Captioning
- 9 May 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 21 (11) , 2942-2956
Abstract
Encoder-decoder models have been widely used in image captioning, and most of them are designed via single Long Short Term Memory (LSTM). The capacity of single-layer network, whose encoder and decoder are integrated together, is limited for such a complex task of image captioning. Moreover, how to effectively increase the “vertical depth” of encoder-decoder remains to be solved. To deal with these problems, a novel Deep Hierarchical Encoder-Decoder Network (DHEDN) is proposed for image captioning, where a deep hierarchical structure is explored to separate the functions of encoder and decoder. This model is capable of efficiently exerting the representation capacity of deep networks to fuse high level semantics of vision and language in generating captions. Specifically, visual representations in top levels of abstraction are simultaneously considered, and each of these levels is associated to one LSTM. The bottom-most LSTM is applied as the encoder of textual inputs. The application of the middle layer in encoder-decoder is to enhance the decoding ability of top-most LSTM. Furthermore, depending on the introduction of semantic enhancement module of image feature and distribution combine module of text feature, variants of architectures of our model are constructed to explore the impacts and mutual interactions among the visual representation, textual representations and the output of the middle LSTM layer. Particularly, the framework is training under a reinforcement learning method to address the exposure bias problem between the training and the testing by the policy gradient optimization. Qualitative analyses indicate the process that our model “translates” image to sentence and further visualization presents the evolution of the hidden states from different hierarchical LSTMs over time. Extensive experiments demonstrate that our model outperforms current state-of-the-art models on three benchmark datasets: Flickr8K, Flickr30K and MSCOCO. On both image captioning and retrieval tasks, our method achieves the best results. On MSCOCO captioning Leaderboard, our method also achieves superior performance.Keywords
Funding Information
- National Natural Science Foundation of China (91646207, 61773377, 61573352)
- Beijing Natural Science Foundation (L172053)
This publication has 44 references indexed in Scilit:
- CIDEr: Consensus-based image description evaluation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Language Models for Image Captioning: The Quirks and What Works2015
- Addressing the Rare Word Problem in Neural Machine Translation2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Every Picture Tells a Story: Generating Sentences from Images2010
- A Feedback Model of Visual AttentionJournal of Cognitive Neuroscience, 2004
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Long Short-Term MemoryNeural Computation, 1997