Abstract
Radar-based human activity sensing possesses the advantage of penetrating detection, making it highly promising for applications in the fields of security, rescue, and medical treatment. With the advancement of deep learning technology, there is a growing interest in the field of human activity sensing with radars, including tasks such as action recognition and pose reconstruction. However, existing research usually treats these two problems as separate tasks. In this work, we propose a dual-task framework for jointly reconstructing human three-dimensional (3D) pose and classifying human action from 4D radar images. Our proposed framework utilizes an ultra-wideband multi-input multi-output (MIMO) radar as the detection sensor to obtain the range-azimuth-height-time 4D imaging data of human targets. A human pose reconstruction network based on 3D convolutional neural network (CNN) is then employed to reconstruct the 3D human pose, while a dual-branch network based on multi-frame 3D human poses and 4D radar image is used to classify the human action. To evaluate the performance of the proposed framework, a dual-task dataset is constructed by merging 4D imaging radar and camera data. Experiments and multi-scenario measurements are constructed to validate the effectiveness of the proposed dual-task framework. The results demonstrate that the proposed dual-task network significantly improves the accuracy of both tasks, while providing a single solution for human action recognition and pose reconstruction.
Funding Information
  • National Natural Science Foundation of China (61971430)

This publication has 28 references indexed in Scilit: