A Multiscale Cross-Modal Interactive Fusion Network for Human Activity Recognition Using Wearable Sensors and Smartphones
- 10 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 11 (16) , 27139-27152
Abstract
Human activity recognition (HAR) enables real-time monitoring of human movement, posture, and activity level, and can provide valuable information for the health management. With the continuous advancement of Internet of Things (IoT) technology, wearable sensors and smartphones equipped with various types of sensors have become widely utilized to collect multimodal data for HAR. However, in multimodal HAR, current fusion methods fall short in capturing intermodality correlations, hampering the full exploitation of complementary information between the modalities and leading to lower recognition accuracy. We thus propose a novel multiscale cross-modal interactive fusion network (MCIFN), which can fully capture correlations between various modalities and obtain an effective fused representation for HAR. Specifically, we employ a multiscale parallel convolution module to extract features from each modality at multiple scales. Then, an interactive fusion strategy based on the cross-modal attention mechanism is introduced to adjust and enhance each modality based on its correlations with other modalities. Additionally, to resolve the information redundancy caused by the interactive fusion strategy, we utilize a hybrid attention module to focus on important information in the fusion representation. Extensive experiments conducted on three publicly available data sets and one private data set demonstrate that our proposed network outperforms the previous baseline networks for HAR. Additionally, our proposed fusion strategy yielded a notable improvement in accuracy ranging from 1.87% to 9.96% compared to existing strategies. These findings imply that our newly proposed network can realize comprehensive multimodal fusion and effectively enhance HAR accuracy, potentially contributing to advancements in individual health management and personalized healthcare interventions.Keywords
Funding Information
- Basic and Applied Basic Research Foundation of Guangdong Province (2021A1515110029)
- National Natural Science Foundation of China (62303496)
This publication has 29 references indexed in Scilit:
- Temporal-channel convolution with self-attention network for human activity recognition using wearable sensorsKnowledge-Based Systems, 2023
- Two-stream transformer network for sensor-based human activity recognitionNeurocomputing, 2022
- ResNet-SE: Channel Attention-Based Deep Residual Network for Complex Activity Recognition Using Wrist-Worn Wearable SensorsIEEE Access, 2022
- DanHAR: Dual Attention Network for multimodal human activity recognition using wearable sensorsApplied Soft Computing, 2021
- Attention induced multi-head convolutional neural network for human activity recognitionApplied Soft Computing, 2021
- A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directionsKnowledge-Based Systems, 2021
- Dilated causal convolution with multi-head self attention for sensor human activity recognitionNeural Computing and Applications, 2021
- Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelinesnpj Digital Medicine, 2020
- Imaging and fusing time series for wearable sensor-based human activity recognitionInformation Fusion, 2020
- mHealthDroid: A Novel Framework for Agile Development of Mobile Health Applications2014