VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
Top Cited Papers
- 17 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (2) , 708-724
Abstract
In this paper, we propose the Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multimodal understanding and generation. Unlike widely-studied vision-language pretraining models, VALOR jointly models the relationships among vision, audio, and language in an end-to-end manner. It consists of three separate encoders for single modality representations and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain the VALOR model: Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language, and audio into the same common space, simultaneously building vision-language, audio-language, and audiovisual-language alignment. MGC learns to generate text tokens under conditions of vision, audio, or both. To promote vision-audio-language pretraining research, we construct a large-scale, high-quality tri-modality dataset named VALOR-1M, containing 1 million audible videos with human-annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and generalize to various downstream tasks (e.g., retrieval, captioning, and question answering) with different input modalities (e.g., vision-language, audio-language, and audiovisual-language). VALOR achieves new state-of-the-art performance on a series of public cross-modality benchmarks.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U21B2043, 62102416)
- Key Research and Development Program of Jiangsu Province (BE2023016-3)
This publication has 78 references indexed in Scilit:
- DaViT: Dual Attention Vision Transformers2022
- MILES: Visual BERT Pre-training with Injected Language Semantics for Video-Text Retrieval2022
- Learning Audio-Video Modalities from Image Captions2022
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval2022
- EclipSE: Efficient Long-Range Video Retrieval Using Sight and Sound2022
- Multi-modal Transformer for Video Retrieval2020
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- UNITER: UNiversal Image-TExt Representation Learning2020
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Microsoft COCO: Common Objects in Context2014