Depth-Aware Sparse Transformer for Video-Language Learning
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4778-4787
Abstract
In Video-Language (VL) learning tasks, a massive amount of text annotations are describing geometrical relationships of instances (e.g. 19.6% to 45.0% in MSVD, MSR-VTT, MSVD-QA and MSVRTT-QA), which often become the bottleneck of the current VL tasks (e.g. 60.8% vs. 98.2% CIDEr in MSVD for geometrical and non-geometrical annotations). Considering the rich spatial information of depth map, an intuitive way is to enrich the conventional 2D visual representations with depth information through current SOTA models, e.g. transformer. However, it is cumbersome to compute the self-attention on a long-range sequence and heterogeneous video-level representations with regard to computation cost and flexibility on various frame scales. To tackle this, we propose a hierarchical transformer, termed Depth-Aware Sparse Transformer (DAST). Specifically, to guarantee computational efficiency, a depth-aware sparse attention modular with linear computational complexity is designed for each transformer layer to learn depth-aware 2D representations. Furthermore, we design a hierarchical structure to maintain multi-scale temporal coherence across long-range dependencies. These qualities of DAST make it compatible with a broad range of video-language tasks, including video captioning (achieving MSVD 107.8%, MSR-VTT 52.5% for CIDEr), video question answering (MSVD-QA 44.1%, MSRVTT-QA 39.4%), and video-text matching (MSR-VTT 215.7 for SumR). Our code is available at https://github.com/zchoi/DASTKeywords
Funding Information
- National Natural Science Foundation of China (No. 62122018, No. 62020106008, No. 61772116, No. 61872064)
- Sichuan Science and Technology Program (No.2022119)
- National Key R&D Program of China (2022YFC2009903/2022YFC2009900)
- Songshan Laboratory (YYJC012022019)
- Fok Ying-Tong Education Foundation (171106)
This publication has 34 references indexed in Scilit:
- Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval2020
- End-to-End Learning of Visual Representations From Uncurated Instructional Videos2020
- Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Location-Aware Graph Convolutional Networks for Video Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Structured Two-Stream Attention Network for Video Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Motion Guided Spatial Attention for Video CaptioningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning2019
- Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?2018
- M3: Multimodal Memory Modelling for Video Captioning2018
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014