Unifying the Video and Question Attentions for Open-Ended Video Question Answering
Authors:
- 29 August 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 26 (12) , 5656-5666
Abstract
Video question answering is an important task toward scene understanding and visual data retrieval. However, current visual question answering works mainly focus on a single static image, which is distinct from the dynamic and sequential visual data in the real world. Their approaches cannot utilize the temporal information in videos. In this paper, we introduce the task of free-form open-ended video question answering. The open-ended answers enable wider applications compared with the common multiple-choice tasks in Visual-QA. We first propose a data set for open-ended Video-QA with the automatic question generation approaches. Then, we propose our sequential video attention and temporal question attention models. These two models apply the attention mechanism on videos and questions, while preserving the sequential and temporal structures of the guides. The two models are integrated into the model of unified attention. After the video and the question are encoded, the answers are generated wordwisely from our models by a decoder. In the end, we evaluate our models on the proposed data set. The experimental results demonstrate the effectiveness of our proposed model.Keywords
Funding Information
- National Basic Research Program of China (973 Program) (2013CB336500)
- National Youth Top-Notch Talent Support Program
This publication has 17 references indexed in Scilit:
- Sequence to Sequence -- Video to Text2015
- Describing Videos by Exploiting Temporal Structure2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- VQA: Visual Question Answering2015
- Show and tell: A neural image caption generator2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Joint Video and Text Parsing for Understanding Events and Answering QueriesIEEE MultiMedia, 2014
- Neural Networks2012
- Long Short-Term MemoryNeural Computation, 1997
- Verbs semantics and lexical selection1994