Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6620-6630
Abstract
Vision-language models (VLMs) pre-trained on large- scale image-text pairs have demonstrated impressive transferability on various visual tasks. Transferring knowledge from such powerful VLMs is a promising direction for building effective video recognition models. However, current exploration in this field is still limited. We believe that the greatest value of pre-trained VLMs lies in building a bridge between visual and textual domains. In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video Attribute Association mechanism, which leverages the Video-to-Text knowledge to generate textual auxiliary attributes for complementing video recognition. ii) We also present a Temporal Concept Spotting mechanism that uses the Text-to-Video expertise to capture temporal saliency in a parameter-free manner, leading to enhanced video representation. Extensive studies on six popular video datasets, including Kinetics-400 & 600, UCF-101, HMDB-51, ActivityNet and Charades, show that our method achieves state-of-the-art performance in various recognition scenarios, such as general, zero-shot, and few-shot video recognition. Our best model achieves a state-of-the-art accuracy of 88.6% on the challenging Kinetics-400 using the released CLIP model. The code is available at https://github.com/whwu95/BIKE.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- STM: SpatioTemporal and Motion Encoding for Action Recognition2019
- I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge GraphsProceedings of the AAAI Conference on Artificial Intelligence, 2019
- ERNIE: Enhanced Language Representation with Informative Entities2019
- A Closer Look at Spatiotemporal Convolutions for Action Recognition2018
- Non-local Neural Networks2018
- A Generative Approach to Zero-Shot and Few-Shot Action Recognition2018
- Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks2017
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era2017
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- HMDB: A large video database for human motion recognition2011