Are Visual-Language Models Effective in Action Recognition? A Comparative Study
Preprint
Abstract
Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- View-Invariant Skeleton Action Representation Learning via Motion RetargetingInternational Journal of Computer Vision, 2024
- Fine-tuned CLIP Models are Efficient Video Learners2023
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection2022
- UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles2021
- Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction2019
- MUREL: Multimodal Relational Reasoning for Visual Question Answering2019
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding2016
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016