Image Caption Generator with Novel Object Injection
Authors:
- 1 December 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
Image captioning is a field within artificial intelligence that is progressing rapidly and it has a lot of potentials. A major problem when working in this field is the limited amount of data that is available to us as is. The only dataset considered suitable enough for the task is the Microsoft: Common Objects in Context (MSCOCO) dataset, which contains about 120,000 training images. This covers about 80 object classes, which is an insufficient amount if we want to create robust solutions that aren't limited to the constraints of the data at hand. In order to overcome this problem, we propose a solution that incorporates Zero-Shot Learning concepts in order to identify unknown objects and classes by using semantic word embeddings and existing state-of-the-art object identification algorithms. Our proposed model, Image Captioning using Novel Word Injection, uses a pre-trained caption generator and works on the output of the generator to inject objects that are not present in the dataset into the caption. We evaluate the model on standardized metrics, namely, BLEU, CIDEr and ROUGE-L. The results, qualitatively and quantitatively, outperform the underlying model.Keywords
This publication has 17 references indexed in Scilit:
- Food Image Recognition Using Very Deep Convolutional Networks2016
- Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data2016
- What Value Do Explicit High Level Concepts Have in Vision to Language Problems?2016
- Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images2015
- Zero-Shot Learning via Semantic Similarity Embedding2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015