More is Better: Precise and Detailed Image Captioning Using Online Positive Recall and Missing Concepts Mining
- 12 July 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (1) , 32-44
Abstract
Recently, a great progress in automatic image captioning has been achieved by using semantic concepts detected from the image. However, we argue that existing concepts-to-caption framework, in which the concept detector is trained using the image-caption pairs to minimize the vocabulary discrepancy, suffers from the deficiency of insufficient concepts. The reasons are two-fold: 1) the extreme imbalance between the number of occurrence positive and negative samples of the concept and 2) the incomplete labeling in training captions caused by the biased annotation and usage of synonyms. In this paper, we propose a method, termed online positive recall and missing concepts mining, to overcome those problems. Our method adaptively re-weights the loss of different samples according to their predictions for online positive recall and uses a two-stage optimization strategy for missing concepts mining. In this way, more semantic concepts can be detected and a high accuracy will be expected. On the caption generation stage, we explore an element-wise selection process to automatically choose the most suitable concepts at each time step. Thus, our method can generate more precise and detailed caption to describe the image. We conduct extensive experiments on the MSCOCO image captioning data set and the MSCOCO online test server, which shows that our method achieves superior image captioning performance compared with other competitive methods.Keywords
Funding Information
- National Natural Science Foundation of China (61572108, 61632007)
- 111 Project (B17008)
This publication has 25 references indexed in Scilit:
- A Distance-Computation-Free Search Scheme for Binary Code DatabasesIEEE Transactions on Multimedia, 2016
- Show and tell: A neural image caption generator2015
- From captions to visual concepts and back2015
- Fully convolutional networks for semantic segmentation2015
- Caffe2014
- Multitask Spectral Clustering by Exploring Intertask CorrelationIEEE Transactions on Cybernetics, 2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Speech recognition with deep recurrent neural networks2013
- Multi-Label ClassificationInternational Journal of Data Warehousing and Mining, 2007
- Long Short-Term MemoryNeural Computation, 1997