A Fine-Grained Spatial-Temporal Attention Model for Video Captioning
Open Access
- 4 November 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 6, 68463-68471
Abstract
Attention mechanism has been extensively used in video captioning tasks, which enables further development of deeper visual understanding. However, most existing video captioning methods apply the attention mechanism on the frame level, which only model the temporal structure and generated words, but ignore the region-level spatial information that provides accurate visual features corresponding to the semantic content. In this paper, we propose a fine-grained spatial-temporal attention model (FSTA), and the spatial information of objects appearing in the video will be our main concern. In the proposed FSTA, we achieve the spatial-hard attention at a fine-grained region level of objects through the mask pooling module and compute the temporal soft attention by using a two-layer LSTM network with attention mechanism to generate sentences. We test the proposed model on two benchmark datasets, namely, MSVD and MSR-VTT. The results indicate that our proposed FSTA model can achieve competitive performance against the state of the arts on both datasets.Keywords
Funding Information
- National Natural Science Foundation of China (61772359, 61472275)
This publication has 30 references indexed in Scilit:
- Semantic Pooling for Complex Event Analysis in Untrimmed VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Bi-Level Semantic Representation Analysis for Multimedia Event DetectionIEEE Transactions on Cybernetics, 2016
- Sequence to Sequence -- Video to Text2015
- Describing Videos by Exploiting Temporal Structure2015
- CIDEr: Consensus-based image description evaluation2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Translating Video Content to Natural Language Descriptions2013
- Relative attributes2011