RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
Authors:
Top Cited Papers
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15460-15469
Abstract
Recent progress on visual question answering has explored the merits of grid features for vision language tasks. Meanwhile, transformer-based models have shown remarkable performance in various sequence prediction problems. However, the spatial information loss of grid features caused by flattening operation, as well as the defect of the transformer model in distinguishing visual words and non visual words, are still left unexplored. In this paper, we first propose Grid-Augmented (GA) module, in which relative geometry features between grids are incorporated to enhance visual representations. Then, we build a BERT-based language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction. To prove the generality of our proposals, we apply the two modules to the vanilla transformer model to build our Relationship-Sensitive Transformer (RSTNet) for image captioning task. The proposed model is tested on the MSCOCO benchmark, where it achieves new state-of-art results on both the Karpathy test split and the online test server. Source code is available at GitHub 1 .Keywords
Funding Information
- National Science Fund for Distinguished Young Scholars
- National Natural Science Foundation of China
- Baidu
This publication has 23 references indexed in Scilit:
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning ChallengeIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Common Subspace for Model and Similarity: Phrase Learning for Caption Generation from Images2015
- Deep visual-semantic alignments for generating image descriptions2015
- Going deeper with convolutions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- BabyTalk: Understanding and Generating Simple Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Every Picture Tells a Story: Generating Sentences from Images2010