Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semisupervised Skeleton-Based Action Recognition
- 10 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (8) , 11035-11048
Abstract
Contrastive learning has been successfully leveraged to learn action representations for addressing the problem of semisupervised skeleton-based action recognition. However, most contrastive learning-based methods only contrast global features mixing spatiotemporal information, which confuses the spatial-and temporal-specific information reflecting different semantic at the frame level and joint level. Thus, we propose a novel spatiotemporal decouple-and-squeeze contrastive learning (SDS-CL) framework to comprehensively learn more abundant representations of skeleton-based actions by jointly contrasting spatial-squeezing features, temporal-squeezing features, and global features. In SDS-CL, we design a new spatiotemporal-decoupling intra–inter attention (SIIA) mechanism to obtain the spatiotemporal-decoupling attentive features for capturing spatiotemporal specific information by calculating spatial-and temporal-decoupling intra-attention maps among joint/motion features, as well as spatial-and temporal-decoupling inter-attention maps between joint and motion features. Moreover, we present a new spatial-squeezing temporal-contrasting loss (STL), a new temporal-squeezing spatial-contrasting loss (TSL), and the global-contrasting loss (GL) to contrast the spatial-squeezing joint and motion features at the frame level, temporal-squeezing joint and motion features at the joint level, as well as global joint and motion features at the skeleton level. Extensive experimental results on four public datasets show that the proposed SDS-CL achieves performance gains compared with other competitive methods.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2018AAA0102001)
- National Natural Science Foundation of China (62222207, 62072245, 62002160, 61932020)
- Natural Science Foundation of Jiangsu Province (BK20211520)
This publication has 73 references indexed in Scilit:
- Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust GatesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Skeleton-Aided Articulated Motion Generation2017
- Enhanced skeleton visualization for view invariant human action recognitionPattern Recognition, 2017
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- A New Representation of Skeleton Sequences for 3D Action Recognition2017
- Interpretable 3D Human Action Analysis with Temporal Convolutional Networks2017
- Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition2016
- Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group2014
- A survey of vision-based methods for action representation, segmentation and recognitionComputer Vision and Image Understanding, 2011
- A survey on vision-based human action recognitionImage and Vision Computing, 2010