Prompt-Guided Prototype-Aware Commonality and Discrimination Learning for Zero-Shot Skeleton-Based Action Recognition
- 21 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 7053-7066
Abstract
Zero-Shot Skeleton-Based Action Recognition (ZSSAR) is an emerging research field focused on developing alignment models that connect skeleton movements with action definitions, thus enabling generalization to unobserved actions. Current methods often employ generative models to reconstruct cross-modal features or enhance mutual information across modalities for alignment. However, when applied to unseen action categories, these models often neglect the inherent consistency among basic actions, thereby diminishing their generalization capabilities. Furthermore, imprecise annotations fail to capture the rich semantic details of actions, resulting in misalignment. Inspired by human cognitive processes and chain of thought, we argue that integrating prior information about human actions with intrinsic commonality knowledge of basic actions is essential for ZSSAR. To actualize this, we propose a novel method termed Prompt-guided Prototype-aware Commonality and Discrimination Learning (PP-CDL). This method utilize the comprehensive world knowledge contained in LLMs, employing tailored prompts to partition seen action categories into distinct, non-overlapping prototype spaces that embody the commonality knowledge of basic actions. Subsequently, we introduce the Inter- and Intra-Prototype Discriminating (I2PD) module and the Intra-Prototype Commonality Mining (IPCM) module. The I2PD amplifies the distinctiveness of knowledge within prototypes, furnishing a personalized search space for the recognition of unseen actions. In contrast, the IPCM models the shared commonality concept within prototypes, bolstering the consistency between skeleton action representations and corresponding text knowledge representations. Experiments on different skeleton action benchmarks demonstrate the significant improvement of our method over existing alternatives.Keywords
Funding Information
- National Natural Science Foundation of China (62222207, 62427808, 62072245)
- Open Foundation of the Key Lab
- Anhui Jianzhu University Anhui Province Key Laboratory of Intelligent Building & Building Energy Saving (IBES2024KF02)
This publication has 44 references indexed in Scilit:
- SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders2024
- Language-guided temporal primitive modeling for skeleton-based action recognitionNeurocomputing, 2024
- Multi-semantic Fusion Model For Generalized Zero-Shot Skeleton-Based Action Recognition2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- 3D Human Action Representation Learning via Cross-View Consistency Pursuit2021
- Skeleton-Based Action Recognition With Shift Graph Convolutional Network2020
- Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders2019
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- 3D skeleton-based human action classification: A surveyPattern Recognition, 2016