Generative Action Description Prompts for Skeleton-based Action Recognition
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10242-10251
Abstract
Skeleton-based action recognition has recently received considerable attention. Current approaches to skeleton-based action recognition are typically formulated as one-hot classification tasks and do not fully exploit the semantic relations between actions. For example, "make victory sign" and "thumb up" are two actions of hand gestures, whose major difference lies in the movement of hands. This information is agnostic from the categorical one-hot encoding of action classes but could be unveiled from the action description. Therefore, utilizing action description in training could potentially benefit representation learning. In this work, we propose a Generative Action-description Prompts (GAP) approach for skeleton-based action recognition. More specifically, we employ a pre-trained large-scale language model as the knowledge engine to automatically generate text descriptions for body parts movements of actions, and propose a multi-modal training scheme by utilizing the text encoder to generate feature vectors for different body parts and supervise the skeleton encoder for action representation learning. Experiments show that our proposed GAP method achieves noticeable improvements over various baseline models without extra computation cost at inference. GAP achieves new state-of-the-arts on popular skeleton-based action recognition benchmarks, including NTU RGB+D, NTU RGB+D 120 and NW-UCLA. The source code is available at https://github.com/MartinXM/GAP.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Zero-Shot Temporal Action Detection via Vision-Language Prompting2022
- Prompting Visual-Language Models for Efficient Video Understanding2022
- MotionCLIP: Exposing Human Motion Generation to CLIP Space2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Spatial-temporal Transformer For Skeleton-based Action Recognition2021
- Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action-Gesture Recognition2021
- Spatial Temporal Transformer Network for Skeleton-Based Action Recognition2021
- Multi-Scale Mixed Dense Graph Convolution Network for Skeleton-Based Action RecognitionIEEE Access, 2021
- Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition2020
- Cross-View Action Modeling, Learning, and Recognition2014