Channel attention and multi-scale graph neural networks for skeleton-based action recognition
Authors:
- 5 September 2022
- journal article
- research article
- Published by SAGE Publications in AI Communications
- Vol. 35 (3) , 187-205
Abstract
3D skeleton data has been widely used in action recognition as the skeleton-based method has achieved good performance in complex dynamic environments. The rise of spatio-temporal graph convolutions has attracted much attention to use graph convolution to extract spatial and temporal features together in the field of skeleton-based action recognition. However, due to the huge difference in the focus of spatial and temporal features, it is difficult to improve the efficiency of extracting the spatiotemporal features. In this paper, we propose a channel attention and multi-scale neural network (CA-MSN) for skeleton-based action recognition with a series of spatio-temporal extraction modules. We exploit the relationship of body joints hierarchically through two modules, i.e., a spatial module which uses the residual GCN network with the channel attention block to extract the high-level spatial features, and a temporal module which uses the multi-scale TCN network to extract the temporal features at different scales. We perform extensive experiments on both the NTU-RGBD60 and NTU-RGBD120 datasets to verify the effectiveness of our network. The comparison results show that our method achieves the state-of-the-art performance with the competitive computing speed. In order to test the application effect of our CA-MSN model, we design a multi-task tandem network consisting of 2D pose estimation, 2D to 3D pose regression and skeleton action recognition model. The end-to-end (RGB video-to-action type) recognition effect is demonstrated. The code is available at https://github.com/Rh-Dang/CA-MSN-action-recognition.git.Keywords
This publication has 56 references indexed in Scilit:
- Enhanced skeleton visualization for view invariant human action recognitionPattern Recognition, 2017
- A New Representation of Skeleton Sequences for 3D Action Recognition2017
- Interpretable 3D Human Action Analysis with Temporal Convolutional Networks2017
- Deeper Depth Prediction with Fully Convolutional Residual Networks2016
- Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition2016
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Going deeper with convolutions2015
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- A “string of feature graphs” model for recognition of complex activities in natural videos2011
- Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interactionProceedings of the IEEE, 2002