Mind's eye: A recurrent visual representation for image caption generation
Authors:
Top Cited Papers
- 1 June 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 2422-2431
Abstract
In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. Critical to our approach is a recurrent neural network that attempts to dynamically build a visual representation of the scene as a caption is being generated or read. The representation automatically learns to remember long-term visual concepts. Our model is capable of both generating novel captions given an image, and reconstructing visual features given an image description. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are equal to or preferred by humans 21.0% of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- CIDEr: Consensus-based image description evaluation2015
- Caffe2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Comparing Automatic Evaluation Measures for Image Description2014
- I2T: Image Parsing to Text DescriptionProceedings of the IEEE, 2010
- Neural Probabilistic Language Models2006
- Long Short-Term MemoryNeural Computation, 1997
- Finding structure in timeCognitive Science, 1990
- Experimental Analysis of the Real-time Recurrent Learning AlgorithmConnection Science, 1989
- Words versus Objects: Comparison of Free Verbal RecallPsychological Reports, 1965