A Position-Aware Transformer for Image Captioning
Open Access
- 1 January 2022
- journal article
- research article
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 70 (1) , 2065-2081
Abstract
Image captioning aims to generate a corresponding description of an image. In recent years, neural encoder-decoder models have been the dominant approaches, in which the Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) are used to translate an image into a natural language description. Among these approaches, the visual attention mechanisms are widely used to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. However, most conventional visual attention mechanisms are based on high-level image features, ignoring the effects of other image features, and giving insufficient consideration to the relative positions between image features. In this work, we propose a Position-Aware Transformer model with image-feature attention and position-aware attention mechanisms for the above problems. The image-feature attention firstly extracts multi-level features by using Feature Pyramid Network (FPN), then utilizes the scaled-dot-product to fuse these features, which enables our model to detect objects of different scales in the image more effectively without increasing parameters. In the position-aware attention mechanism, the relative positions between image features are obtained at first, afterwards the relative positions are incorporated into the original image features to generate captions more accurately. Experiments are carried out on the MSCOCO dataset and our approach achieves competitive BLEU-4, METEOR, ROUGE-L, CIDEr scores compared with some state-of-the-art approaches, demonstrating the effectiveness of our approach.Keywords
This publication has 8 references indexed in Scilit:
- Adam revisited: a weighted past gradients perspectiveFrontiers of Computer Science, 2020
- An Improved Deep Fusion CNN for Image RecognitionComputers, Materials & Continua, 2020
- Predicting Concrete Compressive Strength using Deep Convolutional Neural Network based on Image CharacteristicsComputers, Materials & Continua, 2020
- Corpus Augmentation for Improving Neural Machine TranslationComputers, Materials & Continua, 2020
- Dependency-Based Local Attention Approach To Neural Machine TranslationComputers, Materials & Continua, 2019
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning ChallengeIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- I2T: Image Parsing to Text DescriptionProceedings of the IEEE, 2010