Ensemble Deep Learning for Skeleton-Based Action Recognition Using Temporal Sliding LSTM Networks
Authors:
- Inwoong LeeILInwoong Lee
- Doyoung KimDKDoyoung Kim
- Seoungyoon KangSKSeoungyoon Kang
- Sanghoon LeeSLSanghoon Lee
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1012-1020
Abstract
This paper addresses the problems of feature representation of skeleton joints and the modeling of temporal dynamics to recognize human actions. Traditional methods generally use relative coordinate systems dependent on some joints, and model only the long-term dependency, while excluding short-term and medium term dependencies. Instead of taking raw skeletons as the input, we transform the skeletons into another coordinate system to obtain the robustness to scale, rotation and translation, and then extract salient motion features from them. Considering that Long Shortterm Memory (LSTM) networks with various time-step sizes can model various attributes well, we propose novel ensemble Temporal Sliding LSTM (TS-LSTM) networks for skeleton-based action recognition. The proposed network is composed of multiple parts containing short-term, mediumterm and long-term TS-LSTM networks, respectively. In our network, we utilize an average ensemble among multiple parts as a final feature to capture various temporal dependencies. We evaluate the proposed networks and the additional other architectures to verify the effectiveness of the proposed networks, and also compare them with several other methods on five challenging datasets. The experimental results demonstrate that our network models achieve the state-of-the-art performance through various temporal features. Additionally, we analyze a relation between the recognized actions and the multi-term TS-LSTM features by visualizing the softmax features of multiple parts.Keywords
This publication has 20 references indexed in Scilit:
- Enhanced skeleton visualization for view invariant human action recognitionPattern Recognition, 2017
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Histogram of Oriented Principal Components for Cross-View Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Elastic functional coding of human actions: From vector-fields to latent variables2015
- Going deeper with convolutions2015
- Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group2014
- Group Sparsity and Geometry Constrained Dictionary Learning for Action Recognition from Depth Maps2013
- Learning Actionlet Ensemble for 3D Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Real-time human pose recognition in parts from single depth imagesCommunications of the ACM, 2013
- Mining actionlet ensemble for action recognition with depth cameras2012