Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition
Open Access
Abstract
Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.Keywords
Funding Information
- Air Force Office of Scientific Research (FA9550-21-1-0224)
This publication has 56 references indexed in Scilit:
- Simultaneous Indoor Tracking and Activity Recognition Using Pyroelectric Infrared SensorsSensors, 2017
- Compressive Representation for Device-Free Activity Recognition with Passive RFID Signal StrengthIEEE Transactions on Mobile Computing, 2017
- Multi-sensor fusion in body sensor networks: State-of-the-art and research challengesInformation Fusion, 2017
- ALPAS: Analog-PIR-Sensor-Based Activity Recognition System in Smarthome2017
- DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks2016
- Development of a Sensing Module for Standing and Moving Human Body Using a Shutter and PIR SensorInternational Journal of Multimedia and Ubiquitous Engineering, 2016
- Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelThe Annals of Applied Statistics, 2015
- Movement Detection of Human Body Segments: Passive radio-frequency identification and machine-learning technologiesIEEE Antennas and Propagation Magazine, 2015
- The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem SolutionsInternational Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 1998
- Sensor fusion potential exploitation-innovative architectures and illustrative applicationsProceedings of the IEEE, 1997