VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Preprint
Abstract
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Clover: Towards A Unified Video-Language Alignment and Fusion Model2023
- Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks2023
- All in One: Exploring Unified Video-Language Pre-Training2023
- VindLU: A Recipe for Effective Video-and-Language Pretraining2023
- DaViT: Dual Attention Vision Transformers2022
- Learning Audio-Video Modalities from Image Captions2022
- EclipSE: Efficient Long-Range Video Retrieval Using Sight and Sound2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Movie DescriptionInternational Journal of Computer Vision, 2017
- Microsoft COCO: Common Objects in Context2014