Ferhat Attal

Despite recent progress in integrating Large Language Models (LLMs) to enhance robotic capabilities, key challenges remain—particularly in interpreting perceptions, interacting with humans, and executing tasks in real-world environments. These challenges are largely due to LLM hallucinations and lack of formal semantic knowledge that hinder contextual understanding. To address this issue, we propose a semantic framework that combines an LLM and a commonsense knowledge graph to align the robot’s perceptions with a suitable task execution plan. The framework integrates pre-trained deep learning models for object detection, face identification, speech recognition, an LLM, and a local knowledge graph based on Resource Description Framework (RDF). The knowledge graph including scene and object description, as well spatial knowledge is used to select and execute the suitable task plan for the given context. Experimental results demonstrate the effectiveness of framework’s components in improving the robot’s interaction and and actuation in its environment.

Current deep learning-based activity recognition and anticipation approaches struggle with unlabeled datasets and the lack of commonsense knowledge about previously unseen objects, i.e., unknown objects. This paper proposes a neurosymbolic framework that combines context recognition, incremental learning, and commonsense reasoning to anticipate activities from egocentric vision, while accounting for perceptual uncertainty and the incompleteness of real-world commonsense knowledge. The incremental learning component leverages language models, while Bayesian reasoning enables continuous updates to an ontological knowledge graph with class and affordance descriptions of these unknown objects. These descriptions are derived from visual feature embeddings extracted using a transformer and enriched with knowledge from various multi-modal large language models (MLLMs). Event calculus and answer set programming are used to formalize domain knowledge, enabling probabilistic symbolic reasoning over contextual events to support the prediction of user intentions and the anticipation of both simple and complex activities. A prototype has been tested in a domestic environment using images captured from a head-mounted camera and real-time object and context event detection.

Endowing a companion robot with cognitive abilities to recognize human daily activities, in particular from body skeletons information, is a significant challenge, which needs complex and novel approaches. Recently, most of the proposed approaches exploit the hand-crafted features or the predefined traversal rules techniques to recognize human daily activities from skeleton information, which often lead to the deficit of robustness and generalization. In this work, a novel hybrid framework for human activity-aware robotic system is proposed. In the low-level, a novel Spatio-Temporal Joint based Convolutional Neural Network (STJ-CNN) is proposed to recognize human daily activities in the ambient environments. In the high-level, novel representation and inference services based on Narrative Knowledge Representation Language (NKRL) are proposed to represent and combine the detected human activities with the ambient events, and to infer the semantic context of the detected activity. Empirical experiments on real-world datasets have been conducted, besides an online demonstration created to validate the proposed approach. The final results show that the proposed approach outperforms the baseline models with a significant improvement up to 24% in terms of F-score on DAHLIA dataset.

One of the main objectives of Ambient Assisted Living (AAL) systems is to proactively provide intelligent services to improve the quality of people's lives in terms of autonomy, safety, and well-being. Designing AAL systems that can autonomously monitor human's activities and provide assistance services poses several challenges of which Human Activity Recognition (HAR) which is critically important to adapt the assistance services to the user. In this paper, a robust multi-label HAR framework is proposed. The proposed framework is composed of two main modules: (i) activity classification module and (ii) classification error detection and correction module. In the first module, machine-learning models are used to predict human activities. Since these models may produce predictions with errors, there is a requirement to detect and correct these errors. The classification error detection and correction module is based on two acyclic directed graphical models and operates in two phases: (i) classification error detection and (ii) classification error correction. The proposed framework is evaluated on the Opportunity dataset, a benchmark and unique dataset for multi-label human daily living activity recognition. The obtained results demonstrate the ability of the proposed framework to improve the performances of HAR.

This article presents a machine learning methodology for diagnosing Parkinson’s disease (PD) based on the use of vertical Ground Reaction Forces (vGRFs) data collected from the gait cycle. A classification engine assigns subjects to healthy or Parkinsonian classes. The diagnosis process involves four steps: data pre-processing, feature extraction and selection, data classification and performance evaluation. The selected features are used as inputs of each classifier. Feature selection is achieved through a wrapper approach established using the random forest algorithm. The proposed methodology uses both supervised classification methods including K-nearest neighbour (K-NN), decision tree (DT), random forest (RF), Naïve Bayes (NB), support vector machine (SVM) and unsupervised classification methods such as K-means and the Gaussian mixture model (GMM). To evaluate the effectiveness of the proposed methodology, an online dataset collected within three different studies is used. This data set includes vGRF measurements collected from eight force sensors placed under each foot of the subjects. Ninety-three patients suffering from Parkinson’s disease and 72 healthy subjects participated in the experiments. The obtained performances are compared with respect to various metrics including accuracy, precision, recall and F-measure. The classification performance evaluation is performed using the leave-one-out cross validation. The results demonstrate the ability of the proposed methodology to accurately differentiate between PD subjects and healthy subjects. For the purpose of validation, the proposed methodology is also evaluated with an additional dataset including subjects with neurodegenerative diseases (Amyotrophic Lateral Sclerosis (ALS) and Huntington’s disease (HD)). The obtained results show the effectiveness of the proposed methodology to discriminate PD subjects from subjects with other neurodegenerative diseases with a relatively high accuracy.

Endowing the robotic systems with cognitive capabilities for recognizing daily activities of humans is an important challenge, which requires sophisticated and novel approaches. Most of the proposed approaches explore pattern recognition techniques which are generally based on hand-crafted features or learned features. In this paper, a novel Hierarchal Multichannel Deep Residual Network (HMResNet) model is proposed for robotic systems to recognize daily human activities in the ambient environments. The introduced model is comprised of multilevel fusion layers. The proposed Multichannel 1D Deep Residual Network model is, at the features level, combined with a Bottleneck MLP neural network to automatically extract robust features regardless of the hardware configuration and, at the decision level, is fully connected with an MLP neural network to recognize daily human activities. Empirical experiments on real-world datasets and an online demonstration are used for validating the proposed model. Results demonstrated that the proposed model outperforms the baseline models in daily human activity recognition.

This paper presents a new approach for automatic recognition of gait phases based on the use of an in-shoe pressure measurement system and a Multiple Regression Hidden Markov Model (MRHMM) that takes into account the sequential completion of the gait phases. Recognition of gait phases is formulated as a multiple polynomial regression problem in which each phase, called a segment, is modelled using an appropriate polynomial function. The MRHMM is learned in an unsupervised manner to avoid manual data labelling, which is a laborious, time-consuming task that is subject to potential errors, particularly for large amounts of data. To evaluate the efficiency of the proposed approach, several performance metrics for classification are used: accuracy, F-measure, recall and precision. Experiments conducted with 5 subjects during walking show the potential of the proposed method to recognize gait phases with relatively high accuracy. The proposed approach outperforms standard unsupervised classification methods (GMM, k-Means and HMM) while remaining competitive with respect to standard supervised classification methods (SVM, RF and k-NN).

Driving errors are considered to be the greatest contributory cause in all road accidents and an important contributory cause of most fatal accidents. This is particularly the case for the users of powered two-wheeled vehicles (PTWs), perhaps because PTW riders play a greater role in the control of their vehicles' stability than four-wheeled vehicle drivers. Thus, observing and analyzing the evolution of riders' behavior in a real-life context is an important step in the identification of the road environment characteristics that constitute a risk factor for PTW riders. A relevant research issue in naturalistic studies is related to the detection and identification of critical riding events from among the vast amount of data recorded during the experiment. In this paper, two approaches were used to automatically detect such critical riding events. First, we formalized this problem in terms of detecting changes in the mean and variance of the signals generated by the acceleration and angular velocity sensors. For this purpose, two steps were performed: (1) a data segmentation and feature extraction step in which the multidimensional time series of accelerometer and angular velocity data were segmented and modeled using a Gaussian mixture model with quadratic logistic proportions; and (2) a classification step in which each detected segment was assigned to the appropriate riding sequence, whether “naturalistic” or “critical” (i.e., fall or near fall), using the k-nearest neighbor algorithm. The second approach was based on online fall detection. This methodology used control charts (a multivariate cumulative sum), an approach that has been traditionally employed for sequential detection. These two algorithms were applied to a database composed of data from a real-life driving experiment. The obtained results show the effectiveness of both methodologies.