Cap4Video++: Enhancing Video Understanding With Auxiliary Captions
- 9 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (7) , 5223-5237
Abstract
Understanding videos, especially aligning them with textual data, presents a significant challenge in computer vision. The advent of vision-language models (VLMs) like CLIP has sparked interest in leveraging their capabilities for enhanced video understanding, showing marked advancements in both performance and efficiency. However, current methods often neglect vital user-generated metadata such as video titles. In this paper, we present Cap4Video++, a universal framework that leverages auxiliary captions to enrich video understanding. More recently, we witness the flourishing of large language models (LLMs) like ChatGPT. Cap4Video++ harnesses the synergy of vision-language models (VLMs) and large language models (LLMs) to generate video captions, utilized in three key phases: (i) Input stage employs Semantic Pair Sampling to extract beneficial samples from captions, aiding contrastive learning. (ii) Intermediate stage sees Video-Caption Cross-modal Interaction and Adaptive Caption Selection work together to bolster video and caption representations. (iii) Output stage introduces a Complementary Caption-Text Matching branch, enhancing the primary video branch by improving similarity calculations. Our comprehensive experiments on text-video retrieval and video action recognition across nine benchmarks clearly demonstrate Cap4Video++'s superiority over existing models, highlighting its effectiveness in utilizing automatically generated captions to advance video understanding.Keywords
Funding Information
- Australian Research Council (DP200103223)
- CRC-P Smart Material Recovery Facility
This publication has 67 references indexed in Scilit:
- Transferring Vision-Language Models for Visual Recognition: A Classifier PerspectiveInternational Journal of Computer Vision, 2023
- Frozen CLIP Models are Efficient Video Learners2022
- Prompting Visual-Language Models for Efficient Video Understanding2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval2022
- NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition2022
- Temporal Saliency Query Network for Efficient Video Recognition2022
- Multi-modal Transformer for Video Retrieval2020
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification2018
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016