HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm
Authors:
- 24 October 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10285-10292
Abstract
To fluently collaborate with people, robots need the ability to recognize human activities accurately. Although modern robots are equipped with various sensors, robust human activity recognition (HAR) still remains a challenging task for robots due to difficulties related to multimodal data fusion. To address these challenges, in this work, we introduce a deep neural network-based multimodal HAR algorithm, HAMLET. HAMLET incorporates a hierarchical architecture, where the lower layer encodes spatio-temporal features from unimodal data by adopting a multi-head self-attention mechanism. We develop a novel multimodal attention mechanism for disentangling and fusing the salient unimodal features to compute the multimodal features in the upper layer. Finally, multimodal features are used in a fully connect neural-network to recognize human activities. We evaluated our algorithm by comparing its performance to several state-of-the-art activity recognition algorithms on three human activity datasets. The results suggest that HAMLET outperformed all other evaluated baselines across all datasets and metrics tested, with the highest top-1 accuracy of 95.12% and 97.45% on the UTD-MHAD [1] and the UT-Kinect [2] datasets respectively, and F1-score of 81.52% on the UCSD-MIT [3] dataset. We further visualize the unimodal and multimodal attention maps, which provide us with a tool to interpret the impact of attention mechanisms concerning HAR.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Skeleton Optical Spectra-Based Action Recognition Using Convolutional Neural NetworksIEEE Transactions on Circuits and Systems for Video Technology, 2016
- Human action recognition using RGB-D sensor and deep convolutional neural networks2016
- Movement Coordination in Human–Robot Teams: A Dynamical Systems ApproachIEEE Transactions on Robotics, 2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- DMMs-Based Multiple Features Fusion for Human Action RecognitionInternational Journal of Multimedia Data Engineering and Management, 2015
- UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor2015
- Joint action perception to enable fluent human-robot teamwork2015
- A Method for Automatic Detection of Psychomotor EntrainmentIEEE Transactions on Affective Computing, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- View invariant human action recognition using histograms of 3D joints2012