Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition
Open Access
Abstract
Gesture recognition is an important direction in computer vision research. Information from the hands is crucial in this task. However, current methods consistently achieve attention on hand regions based on estimated keypoints, which will significantly increase both time and complexity, and may lose position information of the hand due to wrong keypoint estimations. Moreover, for dynamic gesture recognition, it is not enough to consider only the attention in the spatial dimension. This paper proposes a multi-scale attention 3D convolutional network for gesture recognition, with a fusion of multimodal data. The proposed network achieves attention mechanisms both locally and globally. The local attention leverages the hand information extracted by the hand detector to focus on the hand region, and reduces the interference of gesture-irrelevant factors. Global attention is achieved in both the human-posture context and the channel context through a dual spatiotemporal attention module. Furthermore, to make full use of the differences between different modalities of data, we designed a multimodal fusion scheme to fuse the features of RGB and depth data. The proposed method is evaluated using the Chalearn LAP Isolated Gesture Dataset and the Briareo Dataset. Experiments on these two datasets prove the effectiveness of our network and show it outperforms many state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundations of China (62002271, 61772396, 61772392, 61902296)
- National Key R&D Program of China (2018YFC0807500)
- Fundamental Research Funds for the Central Universities (XJS210310)
- Province Key R&D Program of Shaanxi (2020LSFP3-15)
- National Natural Science Foundation of Shaanxi Province (2020JQ-330, 2020JM-195)
- China Postdoctoral Science Foundation (2019M663640)
- Guangxi Key Laboratory of Trusted Software (KX202061)
- Key R&D Projects of Qingdao Science and Technology Plan (21-1-2-18-xx)
This publication has 25 references indexed in Scilit:
- Large-scale gesture recognition with a fusion of RGB-D data based on optical flow and the C3D modelPattern Recognition Letters, 2019
- Learning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition2017
- Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature2017
- Large-Scale Gesture Recognition With a Fusion of RGB-D Data Based on Saliency Theory and C3D ModelIEEE Transactions on Circuits and Systems for Video Technology, 2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- ChaLearn Looking at People RGB-D Isolated and Continuous Datasets for Gesture Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- The ChaLearn gesture dataset (CGD 2011)Machine Vision and Applications, 2014
- Novel Haar features for real-time hand gesture recognition using SVMJournal of Real-Time Image Processing, 2012