A Text-Guided Generation and Refinement Model for Image Captioning
- 24 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 2966-2977
Abstract
A high-quality image description requires not only the logic and fluency of language but also the richness and accuracy of the content. However, due to the semantic gap between vision and language, most existing image captioning approaches that directly learn the cross-modal mapping from the vision to language are difficult to meet these two requirements simultaneously. Inspired by the progressive learning mechanism, we trace the generating + refining route and propose a novel Text-Guided Generation and Refinement (dubbed as TGGAR) model with assistance from the guide text to improve the quality of captions. The guide text is selected from the training set according to the visual content similarity, then utilized to explore the salient objects and extend the candidate words. Specifically, we follow the encoder-decoder architecture, and design a Text-Guided Relation Encoder (TGRE) to learn the visual representation that is more consistent with human visual cognition. Besides, we divide the decoder part into two submodules: a Generator for the primary sentence generation and a Refiner for the sentence refinement. Generator, consisting of a standard LSTM and a Gate on Attention (GOA) module, aims to generate the primary sentence logically and fluently. Refiner contains a caption encoder module, an attention-based LSTM and a GOA module, which iteratively modifies the details in the primary caption to make captions rich and accurate with help of the guide text. Extensive experiments on the MS COCO captioning dataset demonstrate that our framework with fewer parameters remains comparable to transformer-based methods, and achieves state-of-the-art performance compared with other relevant approaches.Keywords
Funding Information
- National Natural Science Foundation of China (61932009, 62172138)
- Fundamental Research Funds for the Central Universities (JZ2021HGTB0082)
This publication has 45 references indexed in Scilit:
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Describing Multimedia Content Using Attention-Based Encoder-Decoder NetworksIEEE Transactions on Multimedia, 2015
- Large Scale Retrieval and Generation of Image DescriptionsInternational Journal of Computer Vision, 2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- BLEU2001
- Long Short-Term MemoryNeural Computation, 1997
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992