SkeletonCLIP: Recognizing Skeleton-based Human Actions with Text Prompts
Authors:
- 10 December 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Human action recognition has been a hot research for decades, and mainstream supervised frameworks include a feature extraction backbone and a softmax classifier to predict daily human actions. When the number of classes applied to the dataset changes, we must retrain the classifier on the well-trained backbone. This pipeline restricts the generalization and transfer ability of the model due to an extra training period. Moreover, replacing action labels with simple number labels discards useful semantic information and can only receive a meaningless classifier at last. In this work, we present a model SkeletonCLIP for skeleton-based human action recognition. We add an alternative text encoder to extract semantic information from labels while keeping the original sequence encoder. We use dot production to measure the similarities of sequence-text pairs in place of traditional classifier head and cross-entropy loss. Experiments from three human action datasets show that our framework can reach a higher recognition accuracy with the help of semantic information when training the network from scratch. The code has been shown at eunseo-v/SkeletonCLIP.Keywords
Funding Information
- National Natural Science Foundation of China
This publication has 25 references indexed in Scilit:
- NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity UnderstandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Spatio-Temporal Attention-Based LSTM Networks for 3D Action Recognition and DetectionIEEE Transactions on Image Processing, 2018
- Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM NetworksIEEE Transactions on Image Processing, 2017
- PKU-MMD2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Deep Residual Learning for Image Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Microsoft Kinect Sensor and Its EffectIEEE MultiMedia, 2012
- HMDB: A large video database for human motion recognition2011