Cross-Modality Compensation Convolutional Neural Networks for RGB-D Action Recognition
- 28 April 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (3) , 1498-1509
Abstract
RGB-D-based human action recognition has attracted much attention recently because it can provide more complementary information than a single modality. However, it is difficult for two modalities to effectively learn spatial-temporal information from each other. To facilitate information interaction between different modalities, a cross-modality compensation convolutional neural network (ConvNet) is proposed for human action recognition, which enhances the discriminative ability by jointly learning compensation features from the RGB and depth modalities. Moreover, we design a cross-modality compensation block (CMCB) to extract compensation features from the RGB and depth modalities. Specifically, CMCB is incorporated into two typical network architectures, ResNet and VGG, to verify the ability to improve the performance of our model. The proposed architecture has been evaluated on three challenging datasets: NTU RGB+D 120, THU-READ and PKU-MMD. We experimentally verify that our proposed model with CMCB is effective for different input types, such as pairs of raw images and dynamic images constructed from the entire RGB-D sequence, and the experimental results show that the proposed framework achieves state-of-the-art performance on all three datasets.Keywords
Funding Information
- National Key Research and Development Program of China (2018YFB1308000)
- National Natural Science Funds of China (61772508, U1913202, U1813205, U1713213, 61872083)
- CAS Key Technology Talent Program, Shenzhen Technology Project (JCYJ20180507182610734, JSGG20191129094012321)
This publication has 57 references indexed in Scilit:
- Multimodal Deep Autoencoder for Human Pose RecoveryIEEE Transactions on Image Processing, 2015
- A primal-dual framework for real-time dense RGB-D scene flow2015
- Database Saliency for Fast Image RetrievalIEEE Transactions on Multimedia, 2015
- Image-based 3D Human Pose Recovery by Multi-view Locality Sensitive Sparse RetrievalIEEE Transactions on Industrial Electronics, 2014
- Latent Tensor Transfer Learning for RGB-D Action Recognition2014
- Realistic human action recognition by Fast HOG3D and self-organization feature mapMachine Vision and Applications, 2014
- Learning to Rank Using User Clicks and Visual Features for Image RetrievalIEEE Transactions on Cybernetics, 2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Multi-view Super Vector for Action Recognition2014
- A tutorial on support vector regressionStatistics and Computing, 2004