Clover: Towards A Unified Video-Language Alignment and Fusion Model
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14856-14866
Abstract
Building a universal Video-Language model for solving various video understanding tasks (e.g., text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal, most recent works build the model by stacking uni-modal and cross-modal feature encoders and train it with pair-wise contrastive pre-text tasks. Though offering attractive generality, the resulted models have to compromise between efficiency and performance. They mostly adopt different architectures to deal with different downstream tasks. We find this is because the pair-wise training cannot well align and fuse features from different modalities. We then introduce Clover-a Correlated Video-Language pre-training method-towards a universal Video-Language modelfor solving multiple video understanding tasks with neither performance nor efficiency compromise. It improves cross-modal feature alignment and fusion via a novel tri-modal alignment pre-training task. Additionally, we propose to enhance the tri-modal alignment via incorporating learning from semantic masked samples and a new pair-wise ranking loss. Clover establishes new state-of-the-arts on multiple downstream tasks, including three retrieval tasks for both zero-shot and fine-tuning settings, and eight video question answering tasks. Codes and pre-trained models will be released at https://github.com/LeeYN-43/Clover.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Spatiotemporal-Textual Co-Attention Network for Video Question AnsweringACM Transactions on Multimedia Computing, Communications, and Applications, 2019
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning2018
- Video Question Answering via Gradually Refined Attention over Appearance and Motion2017
- Localizing Moments in Video with Natural Language2017
- Focal Loss for Dense Object Detection2017
- A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering2017
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- NLTK2002