Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data
- 23 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (7) , 4747-4762
Abstract
Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62032006, 62102092)
This publication has 40 references indexed in Scilit:
- Semi-supervised Vision Transformers2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- A Straightforward Framework for Video Retrieval Using CLIP2021
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Transductive Zero-Shot Action Recognition by Word-Vector EmbeddingInternational Journal of Computer Vision, 2017
- Microsoft COCO: Common Objects in Context2014