Abstract
Action recognition in dynamic video remains challenging, particularly when distinguishing between visually similar actions. While existing methods often rely on holistic representations, they overlook the fine-grained details that are significant for accurate classification. We propose a novel Fine-grained Teacher-student CLIP (FT-CLIP) that integrates body part analysis with holistic action recognition through a teacher-student architecture, bridging the gap between fine-grained action parsing and overall action understanding. The teacher model processes individual body parts alongside specialized description to generate part-specific features, which are then aggregated and distilled into the student model. Through knowledge distillation with learnable prompts, the student model effectively learns to capture subtle action distinctions while maintaining efficient inference. FT-CLIP achieves a more nuanced understanding of complex actions by progressing from detailed body part analysis to comprehensive action recognition. Experiments on Kinetics-TPS under a fully-supervised setting and on HMDB51 and UCF101 under a zero-shot setting demonstrate the effectiveness of our method.
Funding Information
  • Natural Science Foundation of Hubei province, China (2023AFB294)
  • National Natural Science Foundation of China (62202345)
  • Open Fund of the Hubei Provincial Engineering Research Center
  • Wuhan Textile University (2023HBITF05)