Motion Guided Spatial Attention for Video Captioning
Authors:
Open Access
- 17 July 2019
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 33 (01) , 8191-8198
Abstract
Sequence-to-sequence models incorporated with attention mechanism have shown promising improvements on video captioning. While there is rich information both inside and between frames, spatial attention is rarely explored and motion information is usually handled by 3D-CNNs as just another modality for fusion. On the other hand, researches about human perception suggest that apparent motion can attract attention. Motivated by this, we aim to learn spatial attention on video frames under the guidance of motion information for caption generation. We present a novel video captioning framework by utilizing Motion Guided Spatial Attention (MGSA). The proposed MGSA exploits the motion between video frames by learning spatial attention from stacked optical flow images with a custom CNN. To further relate the spatial attention maps of video frames, we designed a Gated Attention Recurrent Unit (GARU) to adaptively incorporate previous attention maps. The whole framework can be trained in an end-to-end manner. We evaluate our approach on two benchmark datasets, MSVD and MSR-VTT. The experiments show that our designed model can generate better video representation and state of the art results are obtained under popular evaluation metrics such as BLEU@4, CIDEr, and METEOR.Keywords
This publication has 19 references indexed in Scilit:
- Hierarchical Boundary-Aware Neural Encoder for Video Captioning2017
- Weakly Supervised Dense Video Captioning2017
- Bidirectional Multirate Reconstruction for Temporal Modeling in Videos2017
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning2016
- Going deeper with convolutions2015
- Softening quantization in bag-of-audio-words2014
- YouTube2Text: Recognizing and Describing Arbitrary Activities Using Semantic Hierarchies and Zero-Shot Recognition2013
- Bayesian surprise attracts human attentionVision Research, 2009