From Body Parts to Holistic Action: A Fine-Grained Teacher-Student CLIP for Action Recognition
- 5 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 32 (10709908) , 1336-1340
Abstract
Action recognition in dynamic video remains challenging, particularly when distinguishing between visually similar actions. While existing methods often rely on holistic representations, they overlook the fine-grained details that are significant for accurate classification. We propose a novel Fine-grained Teacher-student CLIP (FT-CLIP) that integrates body part analysis with holistic action recognition through a teacher-student architecture, bridging the gap between fine-grained action parsing and overall action understanding. The teacher model processes individual body parts alongside specialized description to generate part-specific features, which are then aggregated and distilled into the student model. Through knowledge distillation with learnable prompts, the student model effectively learns to capture subtle action distinctions while maintaining efficient inference. FT-CLIP achieves a more nuanced understanding of complex actions by progressing from detailed body part analysis to comprehensive action recognition. Experiments on Kinetics-TPS under a fully-supervised setting and on HMDB51 and UCF101 under a zero-shot setting demonstrate the effectiveness of our method.Keywords
Funding Information
- Natural Science Foundation of Hubei province, China (2023AFB294)
- National Natural Science Foundation of China (62202345)
- Open Fund of the Hubei Provincial Engineering Research Center
- Wuhan Textile University (2023HBITF05)
This publication has 26 references indexed in Scilit:
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models2024
- TCP: Textual-Based Class-Aware Prompt Tuning for Visual-Language Model2024
- Video-Language Graph Convolutional Network for Human Action Recognition2024
- Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models2023
- MaPLe: Multi-modal Prompt Learning2023
- Fine-tuned CLIP Models are Efficient Video Learners2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Conditional Prompt Learning for Vision-Language Models2022
- Digital Twin: Enabling Technologies, Challenges and Open ResearchIEEE Access, 2020
- A review of Convolutional-Neural-Network-based action recognitionPattern Recognition Letters, 2019