Robust Gait Recognition Based on Deep CNNs With Camera and Radar Sensor Fusion
Authors:
- 16 February 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 10 (12) , 10817-10832
Abstract
In recent years, gait recognition has emerged as an important and promising solution for human identification. Generally, gait recognition is based on a single type of sensor, such as a camera or a radar. However, data of a single modality may only capture inadequate gait features of a person, such as camera data lacking the intuitive micro-motion pattern information and radar data lacking the information about gait appearance, making gait-based human identification system vulnerable to complex covariate conditions, e.g., cross-view and cross-walking-condition. To build a robust and reliable gait-based human identification system, in this study, we propose a multisensor gait recognition framework with deep convolutional neural networks (CNNs) by fusing camera gait energy images (GEIs) and radar time-Doppler spectrograms. To learn the fine-grained gait appearance features, we propose a body-part spatial attention (BPSA) module to obtain more discriminative body part representations of GEIs. To learn the gait micro-motion pattern, we propose a long-short temporal relation modeling (LSTRM) module to obtain the local and global micro-motion representation of time-Doppler spectrograms. Finally, we fuse the discriminative body part representation and the micro-motion pattern at the multiscale feature space to obtain richer and more robust gait features for human identification. We provide an extensive empirical evaluation in terms of various complex covariate conditions, namely, cross-view and cross-walking-condition. Experiments on 121 subjects with eight views and three walking conditions of camera and radar data show our proposed method is more robust and accurate.Keywords
Funding Information
- National Science Foundation of China (U21B2039)
- Higher Education Discipline Innovation Project
This publication has 51 references indexed in Scilit:
- A Grassmannian Approach to Address View Change Problem in Gait RecognitionIEEE Transactions on Cybernetics, 2016
- A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Micro-Doppler Feature Extraction Based on Time-Frequency Spectrogram for Ground Moving Targets Classification With Low-Resolution RadarIEEE Sensors Journal, 2016
- View Transformation Model Incorporating Quality Measures for Cross-View Gait RecognitionIEEE Transactions on Cybernetics, 2015
- Knowledge Exploitation for Human Micro-Doppler ClassificationIEEE Geoscience and Remote Sensing Letters, 2015
- Noise Reduction Method Based on Principal Component Analysis With Beta Process for Micro-Doppler Radar SignaturesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015
- Gait recognition using gait Gaussian image2015
- Accelerometer-Based Gait Recognition by Sparse Representation of Signature Points With ClustersIEEE Transactions on Cybernetics, 2014
- Early Hierarchical Contexts Learned by Convolutional Networks for Image Segmentation2014
- Improved gait recognition by gait dynamics normalizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2006