TASTA: Text‐Assisted Spatial and Temporal Attention Network for Video Question Answering
Authors:
Open Access
- 22 February 2023
- journal article
- research article
- Published by Wiley in Advanced Intelligent Systems
- Vol. 5 (4)
Abstract
Video question answering (VideoQA) is a typical task that integrates language and vision. The key for VideoQA is to extract relevant and effective visual information for answering a specific question. Information selection is believed to be necessary for this task due to the large amount of irrelevant information in the video, and explicitly learning an attention model can be a reasonable and effective solution for the selection. Herein, a novel VideoQA model called Text‐Assisted Spatial and Temporal Attention Network (TASTA) is proposed, which shows the great potential of explicitly modeling attention. TASTA is made to be simple, small, clean, and efficient for clear performance justification and possible easy extension. Its success is mainly from two new strategies of better using the textual information. Experimental results on a large and most representative dataset, TGIF‐QA, show the significant superiority of TASTA w.r.t. the state‐of‐the‐art and demonstrate the effectiveness of its key components via ablation studies.Keywords
Funding Information
- National Natural Science Foundation of China (61972016, 62032016)
- Natural Science Foundation of Beijing (L191007)
This publication has 18 references indexed in Scilit:
- Long Short-Term Relation Transformer With Global Gating for Video CaptioningIEEE Transactions on Image Processing, 2022
- Interaction augmented transformer with decoupled decoding for video captioningNeurocomputing, 2022
- VD-PCR: Improving visual dialog with pronoun coreference resolutionPattern Recognition, 2022
- Hierarchical Representation Network With Auxiliary Tasks for Video Captioning and Video Question AnsweringIEEE Transactions on Image Processing, 2021
- Adaptive Spatio-Temporal Graph Enhanced Vision-Language Representation for Video QAIEEE Transactions on Image Processing, 2021
- Smart Manufacturing for Smart Cities—Overview, Insights, and Future DirectionsAdvanced Intelligent Systems, 2020
- Structured Two-Stream Attention Network for Video Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2019
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- No free lunch theorems for optimizationIEEE Transactions on Evolutionary Computation, 1997