Looking Back and Forward: Enhancing Image Captioning with Global Semantic Guidance
Authors:
- 18 July 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
Image captioning requires models to translate an image into natural language description. Recent captioning methods are mostly based on the popular encoder-decoder framework, which uses the encoder to encode an image into a set of vectors and a decoder to produce the caption word-by-word. However, when generating the next word, the decoder only looks back at the preceding words but fails to foresee the posterior ones that are supposed to be produced, lacking awareness of the global semantic information of the image. Moreover, the existing generated caption provides global semantic information of a given image but sometimes contains inconsistent content that needs to be fixed. This paper proposes a novel Temporal-Free Semantic-Guided attention mechanism (TFSG) to utilize the raw caption pre-generated by a primary decoder as the extra input to provide global semantic guidance during generation, deepening visual understanding by balancing the semantic and visual information. Specifically, we design two attention-based structures for TFSG to selectively take the essence from the semantic and visual information to produce refined descriptions. We apply TFSG to two popular captioning models and conduct extensive experiments on the public MS COCO benchmark. Results show that our proposed model achieves superior performance over many state-of-the-art models.Keywords
Funding Information
- National Key Research and Development Program of China (2018YFB1004502)
- National Natural Science Foundation of China (62002373,61690203)
This publication has 26 references indexed in Scilit:
- Label guided correlation hashing for large-scale cross-modal retrievalMultimedia Tools and Applications, 2019
- Exploring Visual Relationship for Image Captioning2018
- Recurrent Fusion Network for Image Captioning2018
- Nonnegative Constrained Graph Based Canonical Correlation Analysis for Multi-view Feature LearningNeural Processing Letters, 2018
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Deep Residual Learning for Image Recognition2016
- Image Captioning with Semantic Attention2016
- Deep visual-semantic alignments for generating image descriptions2015