Mohammed Alarfaj
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Mohammed Alarfaj has a total of 50 co-authors. Here are the top co-authors by number of shared publications:
- Dong-Seong Kim
ICT Convergence Research Center, Department of IT Convergence Engineering, Kumoh National Institute of Technology, Gumi, South Korea
3 shared publications - Shaharyar Kamal
Department of Electrical Engineering, University of Chile, Santiago, Chile
3 shared publications - Yazeed Yasin Ghadi
Department of Computer Science and Software Engineering, Al Ain University, Al Ain, UAE
3 shared publications Ahmad Jalal
Department of Computer Science, Air University, Islamabad, Pakistan
2 shared publicationsHuaping Liu
Department of Electrical and Computer Engineering, Oregon State University, Corvallis, OR 97331
2 shared publications- Huaping Liu
Oregon State University
2 shared publications
Latest Publications
- Open Access
- 10 November 2025
- Published by Frontiers Media SA in Frontiers in Human Neuroscience
- Vol. 19, 1669919
Deep learning approaches for diagnosing seizure based on EEG signal analysis
Introduction: Epilepsy is diagnosed in about 1% of the world’s population as a common brain disease. Timely prediction and detection of seizures can significantly improve the lives of epilepsy patients. Methods: The study has garnered considerable attention over recent years, particularly in the context of advanced computational methods. However, current seizure detection methods still face several limitations, including high inter-patient variability, noisy and non-stationary EEG signals, and the limited generalization ability of single deep learning (DL) models. This paper presents an Ensemble of Deep Transfer Learning (EDTL) models for personalized seizure detection. The technique combines ResNet and EfficientNet methods along with a customized two-Dimensional Convolutional Neural Network (2DCNN) method for patient-specific seizure detection using EEG data. Raw data from the recordings of seizure patients is transformed into EEG signals. Personalized sliding windows are used to extract and store spectrograms for the patients. Patient-specific features are extracted from individual records. EEG signals are normalized for consistent scaling. Short Time Fourier Transform (STFT) is then applied for continuous window slicing over short time intervals. To address the limitations above, the proposed EDTL framework integrates general-purpose pre trained models with a domain-specific custom 2DCNN to capture complementary features. This design improves robustness against noise, enhances adaptability to patient-specific variability, and achieves better generalization compared to individual models. The transformed data is then passed on to train and optimize the models independently and later combined into EDTL. Results and discussion: A comparative evaluation is performed using standard evaluation metrics on two datasets, the CHB-MIT Scalp EEG Database and Turkish Epilepsy EEG Dataset. The proposed EDTL models are evaluated against the individual models on standard performance metrics, with the EDTL achieving the highest performance of 99.23% on the AUC.
- 10 November 2025
- Published by Frontiers Media SA in Frontiers in Human Neuroscience
- Vol. 19, 1669919
Open AccessDeep learning approaches for diagnosing seizure based on EEG signal analysis
Introduction: Epilepsy is diagnosed in about 1% of the world’s population as a common brain disease. Timely prediction and detection of seizures can significantly improve the lives of epilepsy patients. Methods: The study has garnered considerable attention over recent years, particularly in the context of advanced computational methods. However, current seizure detection methods still face several limitations, including high inter-patient variability, noisy and non-stationary EEG signals, and the limited generalization ability of single deep learning (DL) models. This paper presents an Ensemble of Deep Transfer Learning (EDTL) models for personalized seizure detection. The technique combines ResNet and EfficientNet methods along with a customized two-Dimensional Convolutional Neural Network (2DCNN) method for patient-specific seizure detection using EEG data. Raw data from the recordings of seizure patients is transformed into EEG signals. Personalized sliding windows are used to extract and store spectrograms for the patients. Patient-specific features are extracted from individual records. EEG signals are normalized for consistent scaling. Short Time Fourier Transform (STFT) is then applied for continuous window slicing over short time intervals. To address the limitations above, the proposed EDTL framework integrates general-purpose pre trained models with a domain-specific custom 2DCNN to capture complementary features. This design improves robustness against noise, enhances adaptability to patient-specific variability, and achieves better generalization compared to individual models. The transformed data is then passed on to train and optimize the models independently and later combined into EDTL. Results and discussion: A comparative evaluation is performed using standard evaluation metrics on two datasets, the CHB-MIT Scalp EEG Database and Turkish Epilepsy EEG Dataset. The proposed EDTL models are evaluated against the individual models on standard performance metrics, with the EDTL achieving the highest performance of 99.23% on the AUC.
- Open Access
- 27 June 2024
- Published by Frontiers Media SA in Frontiers in Artificial Intelligence
- Vol. 7, 1424190
Wearable sensors based on artificial intelligence models for human activity recognition
Human motion detection technology holds significant potential in medicine, health care, and physical exercise. This study introduces a novel approach to human activity recognition (HAR) using convolutional neural networks (CNNs) designed for individual sensor types to enhance the accuracy and address the challenge of diverse data shapes from accelerometers, gyroscopes, and barometers. Specific CNN models are constructed for each sensor type, enabling them to capture the characteristics of their respective sensors. These adapted CNNs are designed to effectively process varying data shapes and sensor-specific characteristics to accurately classify a wide range of human activities. The late-fusion technique is employed to combine predictions from various models to obtain comprehensive estimates of human activity. The proposed CNN-based approach is compared to a standard support vector machine (SVM) classifier using the one-vs-rest methodology. The late-fusion CNN model showed significantly improved performance, with validation and final test accuracies of 99.35 and 94.83% compared to the conventional SVM classifier at 87.07 and 83.10%, respectively. These findings provide strong evidence that combining multiple sensors and a barometer and utilizing an additional filter algorithm greatly improves the accuracy of identifying different human movement patterns.
- 27 June 2024
- Published by Frontiers Media SA in Frontiers in Artificial Intelligence
- Vol. 7, 1424190
Open AccessWearable sensors based on artificial intelligence models for human activity recognition
Human motion detection technology holds significant potential in medicine, health care, and physical exercise. This study introduces a novel approach to human activity recognition (HAR) using convolutional neural networks (CNNs) designed for individual sensor types to enhance the accuracy and address the challenge of diverse data shapes from accelerometers, gyroscopes, and barometers. Specific CNN models are constructed for each sensor type, enabling them to capture the characteristics of their respective sensors. These adapted CNNs are designed to effectively process varying data shapes and sensor-specific characteristics to accurately classify a wide range of human activities. The late-fusion technique is employed to combine predictions from various models to obtain comprehensive estimates of human activity. The proposed CNN-based approach is compared to a standard support vector machine (SVM) classifier using the one-vs-rest methodology. The late-fusion CNN model showed significantly improved performance, with validation and final test accuracies of 99.35 and 94.83% compared to the conventional SVM classifier at 87.07 and 83.10%, respectively. These findings provide strong evidence that combining multiple sensors and a barometer and utilizing an additional filter algorithm greatly improves the accuracy of identifying different human movement patterns.
- Open Access
- 8 April 2024
- Published by International Association of Online Engineering (IAOE) in International Journal of Engineering Pedagogy (iJEP)
- Vol. 14 (3) , 123-139
Experience of Project-Based Learning: Challenges, Assessment, and Analysis
Though theoretical knowledge is essential, there is a growing belief that when teaching applied sciences, particularly engineering, a great emphasis should be placed on the development of students’ practical skills. Therefore, some courses and curricula need to be revised to enhance the current state of education in this field. This can be achieved through the introduction of project-based learning (PBL). The PBL approach is ideal for teaching engineering disciplines because its framework promotes critical thinking and problem-solving skills, which are essential for a professional engineering career. This paper explains the implementation and comparative analysis of the PBL method in the departments of Electrical Engineering and Biomedical Engineering at King Faisal University. The challenges faced in this implementation include incorporating the PBL approach into the departments’ overall instructional technique, defining the roles of PBL coordinators, and addressing the characteristics of the PBL problems that arise in the courses offered. Furthermore, the implementation of PBL projects is evaluated based on the common courses offered in both departments. It is taken as an example and assessed based on problem-solving, teamwork skills, and the outcomes of accreditation board for engineering and technology. The results of this implementation are reported in this paper. Furthermore, we actively shared our experience managing course projects remotely during the fall semester, which COVID-19 impacted. Recommendations for future work are discussed, emphasizing the importance of having sufficient resources and fostering collaboration with industry and researchers from diverse disciplines.
- 8 April 2024
- Published by International Association of Online Engineering (IAOE) in International Journal of Engineering Pedagogy (iJEP)
- Vol. 14 (3) , 123-139
Open AccessExperience of Project-Based Learning: Challenges, Assessment, and Analysis
Though theoretical knowledge is essential, there is a growing belief that when teaching applied sciences, particularly engineering, a great emphasis should be placed on the development of students’ practical skills. Therefore, some courses and curricula need to be revised to enhance the current state of education in this field. This can be achieved through the introduction of project-based learning (PBL). The PBL approach is ideal for teaching engineering disciplines because its framework promotes critical thinking and problem-solving skills, which are essential for a professional engineering career. This paper explains the implementation and comparative analysis of the PBL method in the departments of Electrical Engineering and Biomedical Engineering at King Faisal University. The challenges faced in this implementation include incorporating the PBL approach into the departments’ overall instructional technique, defining the roles of PBL coordinators, and addressing the characteristics of the PBL problems that arise in the courses offered. Furthermore, the implementation of PBL projects is evaluated based on the common courses offered in both departments. It is taken as an example and assessed based on problem-solving, teamwork skills, and the outcomes of accreditation board for engineering and technology. The results of this implementation are reported in this paper. Furthermore, we actively shared our experience managing course projects remotely during the fall semester, which COVID-19 impacted. Recommendations for future work are discussed, emphasizing the importance of having sufficient resources and fostering collaboration with industry and researchers from diverse disciplines.
- Open Access
- 4 January 2024
- Published by Elsevier in Case Studies in Construction Materials
- Vol. 20, e02836
Machine learning based prediction models for spilt tensile strength of fiber reinforced recycled aggregate concrete
- 4 January 2024
- Published by Elsevier in Case Studies in Construction Materials
- Vol. 20, e02836
Open AccessMachine learning based prediction models for spilt tensile strength of fiber reinforced recycled aggregate concrete
- Open Access
- 9 November 2022
- Published by MDPI AG in Diagnostics
- Vol. 12 (11) , 2739
AAL and Internet of Medical Things for Monitoring Type-2 Diabetic Patients
Due to an aging population, assisted-care options are required so that senior citizens may maintain their independence at home for a longer time and rely less on caretakers. Ambient Assisted Living (AAL) encourages the creation of solutions that can help to optimize the environment for senior citizens with assistance while greatly reducing their challenges. A framework based on the Internet of Medical Things (IoMT) is used in the current study for the implementation of AAL technology to help patients with Type-2 diabetes. A glucose oxide sensor is used to monitor diabetic elderly people continuously. Spectrogram images are created from the recorded data from the sensor to assess and detect aberrant glucose levels. DenseNet-169 examines and analyzes the spectrogram pictures, and messages are sent to caregivers when aberrant glucose levels are detected. The current work describes both the spectrogram image analysis and the signal-to-spectrogram generating method. The study presents a future perspective model for a mobile application for real-time patient monitoring. Benchmark metrics evaluate the application’s performances, including sensitivity, specificity, accuracy, and F1-score. Several cross--validations are used to evaluate the model’s performance. The findings demonstrate that the proposed model can correctly identify patients with abnormal blood glucose levels.
- 9 November 2022
- Published by MDPI AG in Diagnostics
- Vol. 12 (11) , 2739
Open AccessAAL and Internet of Medical Things for Monitoring Type-2 Diabetic Patients
Due to an aging population, assisted-care options are required so that senior citizens may maintain their independence at home for a longer time and rely less on caretakers. Ambient Assisted Living (AAL) encourages the creation of solutions that can help to optimize the environment for senior citizens with assistance while greatly reducing their challenges. A framework based on the Internet of Medical Things (IoMT) is used in the current study for the implementation of AAL technology to help patients with Type-2 diabetes. A glucose oxide sensor is used to monitor diabetic elderly people continuously. Spectrogram images are created from the recorded data from the sensor to assess and detect aberrant glucose levels. DenseNet-169 examines and analyzes the spectrogram pictures, and messages are sent to caregivers when aberrant glucose levels are detected. The current work describes both the spectrogram image analysis and the signal-to-spectrogram generating method. The study presents a future perspective model for a mobile application for real-time patient monitoring. Benchmark metrics evaluate the application’s performances, including sensitivity, specificity, accuracy, and F1-score. Several cross--validations are used to evaluate the model’s performance. The findings demonstrate that the proposed model can correctly identify patients with abnormal blood glucose levels.
- 22 August 2022
- Published by Springer Nature in Journal of Electrical Engineering & Technology
- Vol. 18 (1) , 651-661
Automatic Fiducial Points Detection for Multi-facial Expressions via Invariant Features and Multi-layer Kernel Sliding Perceptron
- 22 August 2022
- Published by Springer Nature in Journal of Electrical Engineering & Technology
- Vol. 18 (1) , 651-661
Automatic Fiducial Points Detection for Multi-facial Expressions via Invariant Features and Multi-layer Kernel Sliding Perceptron
- Open Access
- 1 March 2022
- Published by SAGE Publications in Measurement and Control
- Vol. 55 (3-4) , 136-145
Arduino Uno Wi-Fi DeMilitarized Zone-based monitoring of solar photovoltaic systems
Due to global warming, governments around the world are focusing on renewable energy as the main source of energy. Gulf countries in particular, are focusing on solar energy due to its abundance and the gulf geolocation that allows the installation of ambitious solar photovoltaic (PV) energy systems and replace conventional power plants. In order to maximize energy extraction from the sun, solar power operators need to constantly monitor key solar PV parameters such as panel temperature, solar irradiation, voltage and current in order to detect abnormalities in the PV system operation and take the necessary actions to keep the PV system performance at its maximum. However, monitoring such parameters on-site is exhausting and expensive. Moreover, remote monitoring for solar PV systems is a more attractive solution once the data collection, processing and transmission become faster and less expensive. This paper describes the design of a low-cost remote monitoring system intended for off-grid solar PV systems. The designed remote monitoring system uses a combination of an Arduino Uno Wi-Fi demilitarized zone (DMZ) communication system with router-configured network sensors calibrated to fulfill the IEC-61,724 standards and it provides meteorological and electrical measurements with an error of 2% or less. There are four primary features of the proposed monitoring system. First, it combines the Arduino Uno Wi-Fi with a DMZ configuration in the router. Second, it complies with the minimum current, voltage, power, and PV-module temperature values set by the IEC-61,724 in such a way that the average error does not exceed the 4% threshold. Third, it provides an accuracy level of 98% for the meteorological and electrical parameters. This level of accuracy exceeds that of commercial data logging equipment. Finally, its total cost is less than $100, which is less than that of available commercial solar PV monitoring systems. Additional features include the ability to select the number of processed measurements per hour and innovative processing, display, and access schemes. These features enable the accessibility of the data from mobile phones and computers within the campus for teaching and experimentation purposes.
- 1 March 2022
- Published by SAGE Publications in Measurement and Control
- Vol. 55 (3-4) , 136-145
Open AccessArduino Uno Wi-Fi DeMilitarized Zone-based monitoring of solar photovoltaic systems
Due to global warming, governments around the world are focusing on renewable energy as the main source of energy. Gulf countries in particular, are focusing on solar energy due to its abundance and the gulf geolocation that allows the installation of ambitious solar photovoltaic (PV) energy systems and replace conventional power plants. In order to maximize energy extraction from the sun, solar power operators need to constantly monitor key solar PV parameters such as panel temperature, solar irradiation, voltage and current in order to detect abnormalities in the PV system operation and take the necessary actions to keep the PV system performance at its maximum. However, monitoring such parameters on-site is exhausting and expensive. Moreover, remote monitoring for solar PV systems is a more attractive solution once the data collection, processing and transmission become faster and less expensive. This paper describes the design of a low-cost remote monitoring system intended for off-grid solar PV systems. The designed remote monitoring system uses a combination of an Arduino Uno Wi-Fi demilitarized zone (DMZ) communication system with router-configured network sensors calibrated to fulfill the IEC-61,724 standards and it provides meteorological and electrical measurements with an error of 2% or less. There are four primary features of the proposed monitoring system. First, it combines the Arduino Uno Wi-Fi with a DMZ configuration in the router. Second, it complies with the minimum current, voltage, power, and PV-module temperature values set by the IEC-61,724 in such a way that the average error does not exceed the 4% threshold. Third, it provides an accuracy level of 98% for the meteorological and electrical parameters. This level of accuracy exceeds that of commercial data logging equipment. Finally, its total cost is less than $100, which is less than that of available commercial solar PV monitoring systems. Additional features include the ability to select the number of processed measurements per hour and innovative processing, display, and access schemes. These features enable the accessibility of the data from mobile phones and computers within the campus for teaching and experimentation purposes.
- Open Access
- 25 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 23964-23979
MS-DLD: Multi-Sensors Based Daily Locomotion Detection via Kinematic-Static Energy and Body-Specific HMMs
More adaptable and user-independent techniques are required for multi-sensors based daily locomotion detection (MS-DLD). This research study proposes a couple of locomotion detection methods using body-worn multi-sensors to successfully categorize several locomotion transitions, including falling, walking, jogging, and jumping, along with body-specific sensors based on the modified hidden Markov models (HMMs) approach. This research presents both standard and state-of-the-art methods for MS-DLD. Conventionally, to improve MS-DLD process, the proposed methodology consists of a wavelet transformed Quaternion-based filter for the inertial signals, patterns recognition in the form of kinematic-static energies, and state-of-the-art multi-features extraction. These features include entropy, spectral, and cepstral coefficients domains. Then, fuzzy logic-based optimization has been introduced in order to achieve the selective features by converting them into codewords. This paper also introduces another state-of-the-art way to model daily locomotion detection and derives body-specific modified HMMs. The model divides the sensor data into three active body-specific parts including head sensors, mid-body sensors, and lower body sensors. Body-specific modified HMMs have been provided with raw data for the three active body-specific sensors and gave better results with less computational complexities when compared to the conventional methods. The proposed systems have been experimentally assessed and trialed over three diverse publicly available datasets: the UP-Fall dataset consisting of falling and other daily life activities, the IM-WSHA dataset comprising everyday locomotion actions, and the ENABL3S gait and locomotion dataset consisting of multiple gait movements. Experimental outcomes indicate that the proposed conventional technique achieved improved results and outperformed existing systems based on detection accuracies of 90.0% and 87.5% over UP-Fall, 86.0% and 88.3% over IM-WSHA, and 86.7% and 90.0% over ENABL3S datasets for kinematic and static energy patterns, respectively. Further, the results show that the state-of-the-art body-specific modified HMMs method achieved 94.3% and 95.0% over UP-Fall, 92.0% and 93.3% over IM-WSHA, and 90.0% and 95.0% over ENABL3S datasets for kinematic and static patterned signals, respectively. The results of state-of-the-art efficient system show a significant increase in detection accuracy when compared to standard systems.
- 25 February 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 23964-23979
Open AccessMS-DLD: Multi-Sensors Based Daily Locomotion Detection via Kinematic-Static Energy and Body-Specific HMMs
More adaptable and user-independent techniques are required for multi-sensors based daily locomotion detection (MS-DLD). This research study proposes a couple of locomotion detection methods using body-worn multi-sensors to successfully categorize several locomotion transitions, including falling, walking, jogging, and jumping, along with body-specific sensors based on the modified hidden Markov models (HMMs) approach. This research presents both standard and state-of-the-art methods for MS-DLD. Conventionally, to improve MS-DLD process, the proposed methodology consists of a wavelet transformed Quaternion-based filter for the inertial signals, patterns recognition in the form of kinematic-static energies, and state-of-the-art multi-features extraction. These features include entropy, spectral, and cepstral coefficients domains. Then, fuzzy logic-based optimization has been introduced in order to achieve the selective features by converting them into codewords. This paper also introduces another state-of-the-art way to model daily locomotion detection and derives body-specific modified HMMs. The model divides the sensor data into three active body-specific parts including head sensors, mid-body sensors, and lower body sensors. Body-specific modified HMMs have been provided with raw data for the three active body-specific sensors and gave better results with less computational complexities when compared to the conventional methods. The proposed systems have been experimentally assessed and trialed over three diverse publicly available datasets: the UP-Fall dataset consisting of falling and other daily life activities, the IM-WSHA dataset comprising everyday locomotion actions, and the ENABL3S gait and locomotion dataset consisting of multiple gait movements. Experimental outcomes indicate that the proposed conventional technique achieved improved results and outperformed existing systems based on detection accuracies of 90.0% and 87.5% over UP-Fall, 86.0% and 88.3% over IM-WSHA, and 86.7% and 90.0% over ENABL3S datasets for kinematic and static energy patterns, respectively. Further, the results show that the state-of-the-art body-specific modified HMMs method achieved 94.3% and 95.0% over UP-Fall, 92.0% and 93.3% over IM-WSHA, and 90.0% and 95.0% over ENABL3S datasets for kinematic and static patterned signals, respectively. The results of state-of-the-art efficient system show a significant increase in detection accuracy when compared to standard systems.
- Open Access
- 1 January 2022
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 72 (1) , 1-18
Human Pose Estimation and Object Interaction for Sports Behaviour
In the new era of technology, daily human activities are becoming more challenging in terms of monitoring complex scenes and backgrounds. To understand the scenes and activities from human life logs, human-object interaction (HOI) is important in terms of visual relationship detection and human pose estimation. Activities understanding and interaction recognition between human and object along with the pose estimation and interaction modeling have been explained. Some existing algorithms and feature extraction procedures are complicated including accurate detection of rare human postures, occluded regions, and unsatisfactory detection of objects, especially small-sized objects. The existing HOI detection techniques are instance-centric (object-based) where interaction is predicted between all the pairs. Such estimation depends on appearance features and spatial information. Therefore, we propose a novel approach to demonstrate that the appearance features alone are not sufficient to predict the HOI. Furthermore, we detect the human body parts by using the Gaussian Matric Model (GMM) followed by object detection using YOLO. We predict the interaction points which directly classify the interaction and pair them with densely predicted HOI vectors by using the interaction algorithm. The interactions are linked with the human and object to predict the actions. The experiments have been performed on two benchmark HOI datasets demonstrating the proposed approach.
- 1 January 2022
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 72 (1) , 1-18
Open AccessHuman Pose Estimation and Object Interaction for Sports Behaviour
In the new era of technology, daily human activities are becoming more challenging in terms of monitoring complex scenes and backgrounds. To understand the scenes and activities from human life logs, human-object interaction (HOI) is important in terms of visual relationship detection and human pose estimation. Activities understanding and interaction recognition between human and object along with the pose estimation and interaction modeling have been explained. Some existing algorithms and feature extraction procedures are complicated including accurate detection of rare human postures, occluded regions, and unsatisfactory detection of objects, especially small-sized objects. The existing HOI detection techniques are instance-centric (object-based) where interaction is predicted between all the pairs. Such estimation depends on appearance features and spatial information. Therefore, we propose a novel approach to demonstrate that the appearance features alone are not sufficient to predict the HOI. Furthermore, we detect the human body parts by using the Gaussian Matric Model (GMM) followed by object detection using YOLO. We predict the interaction points which directly classify the interaction and pair them with densely predicted HOI vectors by using the interaction algorithm. The interactions are linked with the human and object to predict the actions. The experiments have been performed on two benchmark HOI datasets demonstrating the proposed approach.
- Open Access
- 25 November 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 167434-167446
An LSTM-Based Approach for Understanding Human Interactions Using Hybrid Feature Descriptors Over Depth Sensors
Over the past few years, automatic recognition of human interactions has drawn significant attention from researchers working in the field of Artificial Intelligence (AI). And feature extraction is one of the most critical tasks in developing efficient Human Interaction Recognition (HIR) systems. Moreover, recent researches in computer vision suggest that robust features lead to higher recognition accuracies. Hence, an improved HIR system has been proposed in this paper that combines 2D and 3D features extracted using machine learning and deep learning techniques. These discriminative features result in accurate classification and help avoid misclassification of similar interactions. Ten keyframes have been extracted from each video to reduce computational complexity. Next, these frames have been preprocessed using image normalization and noise removal techniques. The Region Of Interest (ROI), which contains the two humans involved in the interaction, has been extracted using motion detection. Then, the human silhouettes have been segmented using the GrabCut algorithm. Next, the extracted silhouettes have been converted into 3D meshes and their heat kernel signatures (HKS) have been obtained to extract key body points. A Convolutional Neural Network (CNN) has been used to extract full-body features from 2D full-body silhouettes. Then, topological and geometric features have been extracted from the key body points. Finally, the combined feature vector has been fed into Long Short-Term Memory (LSTM) and each interaction has been recognized using a Softmax classifier. The proposed system has been validated via extensive experimentation on three challenging RGB+D datasets. The recognition accuracies of 91.63%, 90.54%, and 90.13% have been achieved with the SBU Kinect Interaction, NTU RGB+D, and ISR-UoL 3D social activity datasets respectively. The results of extensive experiments performed on the proposed system suggest that it can be used effectively for various applications, such as security, surveillance, health monitoring, and assisted living.
- 25 November 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 167434-167446
Open AccessAn LSTM-Based Approach for Understanding Human Interactions Using Hybrid Feature Descriptors Over Depth Sensors
Over the past few years, automatic recognition of human interactions has drawn significant attention from researchers working in the field of Artificial Intelligence (AI). And feature extraction is one of the most critical tasks in developing efficient Human Interaction Recognition (HIR) systems. Moreover, recent researches in computer vision suggest that robust features lead to higher recognition accuracies. Hence, an improved HIR system has been proposed in this paper that combines 2D and 3D features extracted using machine learning and deep learning techniques. These discriminative features result in accurate classification and help avoid misclassification of similar interactions. Ten keyframes have been extracted from each video to reduce computational complexity. Next, these frames have been preprocessed using image normalization and noise removal techniques. The Region Of Interest (ROI), which contains the two humans involved in the interaction, has been extracted using motion detection. Then, the human silhouettes have been segmented using the GrabCut algorithm. Next, the extracted silhouettes have been converted into 3D meshes and their heat kernel signatures (HKS) have been obtained to extract key body points. A Convolutional Neural Network (CNN) has been used to extract full-body features from 2D full-body silhouettes. Then, topological and geometric features have been extracted from the key body points. Finally, the combined feature vector has been fed into Long Short-Term Memory (LSTM) and each interaction has been recognized using a Softmax classifier. The proposed system has been validated via extensive experimentation on three challenging RGB+D datasets. The recognition accuracies of 91.63%, 90.54%, and 90.13% have been achieved with the SBU Kinect Interaction, NTU RGB+D, and ISR-UoL 3D social activity datasets respectively. The results of extensive experiments performed on the proposed system suggest that it can be used effectively for various applications, such as security, surveillance, health monitoring, and assisted living.