Dual-Task Human Activity Sensing for Pose Reconstruction and Action Recognition Using 4-D Imaging Radar
- 30 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 23 (19) , 23927-23940
Abstract
Radar-based human activity sensing possesses the advantage of penetrating detection, making it highly promising for applications in the fields of security, rescue, and medical treatment. With the advancement of deep learning technology, there is a growing interest in the field of human activity sensing with radars, including tasks such as action recognition and pose reconstruction. However, existing research usually treats these two problems as separate tasks. In this work, we propose a dual-task framework for jointly reconstructing human three-dimensional (3D) pose and classifying human action from 4D radar images. Our proposed framework utilizes an ultra-wideband multi-input multi-output (MIMO) radar as the detection sensor to obtain the range-azimuth-height-time 4D imaging data of human targets. A human pose reconstruction network based on 3D convolutional neural network (CNN) is then employed to reconstruct the 3D human pose, while a dual-branch network based on multi-frame 3D human poses and 4D radar image is used to classify the human action. To evaluate the performance of the proposed framework, a dual-task dataset is constructed by merging 4D imaging radar and camera data. Experiments and multi-scenario measurements are constructed to validate the effectiveness of the proposed dual-task framework. The results demonstrate that the proposed dual-task network significantly improves the accuracy of both tasks, while providing a single solution for human action recognition and pose reconstruction.Keywords
Funding Information
- National Natural Science Foundation of China (61971430)
This publication has 28 references indexed in Scilit:
- Non-Contact Human Motion Recognition Based on UWB RadarIEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2018
- Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image2017
- A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences2017
- Miniaturized Millimeter-Wave Radar Sensor for High-Accuracy ApplicationsIEEE Transactions on Microwave Theory and Techniques, 2017
- Action recognition using micro-Doppler signatures and a recurrent neural network2017
- Deep Residual Learning for Image Recognition2016
- Human Detection and Activity Classification Based on Micro-Doppler Signatures Using Deep Convolutional Neural NetworksIEEE Geoscience and Remote Sensing Letters, 2015
- Capturing the human figure through a wallACM Transactions on Graphics, 2015
- Focusing of synthetic aperture ultra wideband data1991
- Image Reconstruction from Projections1984