Re-Caption: Saliency-Enhanced Image Captioning Through Two-Phase Learning
- 17 July 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 29 (10577149) , 694-709
Abstract
Visual and semantic saliency are important in image captioning. However, single-phase image captioning benefits little from limited saliency without a saliency predictor. In this paper, a novel saliency-enhanced re-captioning framework via two-phase learning is proposed to enhance the single-phase image captioning. In the framework, visual saliency and semantic saliency are distilled from the first-phase model and fused with the second-phase model for model self-boosting. The visual saliency mechanism can generate a saliency map and a saliency mask for an image without learning a saliency map predictor. The semantic saliency mechanism sheds some lights on the properties of words with part-of-speech Noun in a caption. Besides, another type of saliency, sample saliency is proposed to explicitly compute the saliency degree of each sample, which helps for more robust image captioning. In addition, how to combine the above three types of saliency for further performance boost is also examined. Our framework can treat an image captioning model as a saliency extractor, which may benefit other captioning models and related tasks. The experimental results on both the Flickr30k and MSCOCO datasets show that the saliency-enhanced models can obtain promising performance gains.Keywords
Funding Information
- National Natural Science Foundation of China (61572140, 61976057)
- Shanghai Municipal R&D Foundation (17DZ1100504, 16JC1420401)
- Natural Science Foundation of Shanghai Municipality (19ZR1417200)
- Humanities and Social Sciences Planning Fund of Ministry of education of China (19YJA630116)
- University of Iowa
This publication has 44 references indexed in Scilit:
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning ChallengeIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Guiding the Long-Short Term Memory Model for Image Caption Generation2015
- Salient Object Detection: A BenchmarkIEEE Transactions on Image Processing, 2015
- Deep visual-semantic alignments for generating image descriptions2015
- SALICON: Saliency in Context2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Boosting and Additive Trees2008
- Long Short-Term MemoryNeural Computation, 1997