Satoshi Endo

Our results reveal that interacting humans inconspicuously modulate muscle activation to extract accurate information about the common target while considering their own and the partner’s sensorimotor noise. A novel computational model was developed to decipher the underlying mechanism: muscle coactivation is adapted to combine haptic information from the interaction with the partner and own visual information in a stochastically optimal manner. This improves the prediction of the target position with minimal metabolic cost in each partner, resulting in the lead of the partner with the most accurate visual information.

During the last few decades, great research endeavors have been applied to healthcare robots, aiming to develop companions that extend the independent living of elderly people. To deploy such robots into the market, it is expected that certain applications should be addressed with repeatability and robustness. Such application is the assistance with medication-related activity, a common need for the majority of elderly people, referred from here on as medication adherence. This paper presents a novel and complete pipeline for assistance provision in monitoring and serving of medication, using a mobile manipulator embedded with action, perception and cognition skills. The challenges tackled in this work comprise, among others, that the robot locates the medication box placed in challenging spots by applying vision based strategies, thus enabling robust grasping. The grasping is performed with strategies that allow environmental contact, accommodated by the manipulator’s admittance controller which offers compliance behavior during interaction with the environment. Robot navigation is applied for the medication delivery, which, combined with active vision methods, enables the automatic selection of parking positions, allowing efficient interaction and monitoring of medication intake activity. The robot skills are orchestrated by a partially observable Markov decision process mechanism which is coupled with a task planner. This enables assistance scenario guidance and offers repeatability as well as gentle degradation of the system upon a failure, thus avoiding uncomfortable situations during human–robot interaction. Experiments have been conducted on the full pipeline, including robot’s deployment in 12 real house environments with real participants that led to very promising results with valuable findings for similar future applications.

Patients with advanced Parkinson’s disease regularly experience unstable motor states. Objective and reliable monitoring of these fluctuations is an unmet need. We used deep learning to classify motion data from a single wrist-worn IMU sensor recording in unscripted environments. For validation purposes, patients were accompanied by a movement disorder expert, and their motor state was passively evaluated every minute. We acquired a dataset of 8,661 minutes of IMU data from 30 patients, with annotations about the motor state (OFF,ON, DYSKINETIC) based on MDS-UPDRS global bradykinesia item and the AIMS upper limb dyskinesia item. Using a 1-minute window size as an input for a convolutional neural network trained on data from a subset of patients, we achieved a three-class balanced accuracy of 0.654 on data from previously unseen subjects. This corresponds to detecting the OFF, ON, or DYSKINETIC motor state at a sensitivity/specificity of 0.64/0.89, 0.67/0.67 and 0.64/0.89, respectively. On average, the model outputs were highly correlated with the annotation on a per subject scale (r = 0.83/0.84; p < 0.0001), and sustained so for the highly resolved time windows of 1 minute (r = 0.64/0.70; p < 0.0001). Thus, we demonstrate the feasibility of long-term motor-state detection in a free-living setting with deep learning using motion data from a single IMU.

Dyadic interaction between humans has gained great research interest in the last years. The effects of factors that influence the interaction, as e.g. roles or skill level matching, are still not well understood. In this paper, we further investigated the effect of skill level matching between partners on learning of a visuo-motor task. Understanding the effect of skill level matching is crucial for applications in collaborative rehabilitation. Fifteen healthy participants were asked to trace a path while being subjected to a visuo-motor rotation (Novice). The Novices were paired with a partner, forming one of the three Dyad Types: a) haptic connection to another Novice, b) haptic connection to an Expert (no visuo-motor rotation), or c) no haptic. The intervention consisted of a Familiarization phase, followed by a Training phase, in which the Novices were learning the task in the respective Dyad Type, and a Test phase in which the learning was assessed (haptic connection removed, if any). Results suggest that learning of the task with a haptic connection to an Expert was least beneficial. However, during the Training phase the dyads comprising an Expert clearly outperformed the dyads with matched skill levels. The results point towards the same direction as previous findings in literature and can be explained by current motor-learning theories. Future work needs to corroborate these preliminary results.

Objective: Modeling of human motor intention plays an essential role in predictively controlling a robotic system in human-robot interaction tasks. In most machine learning techniques, human motor behavior is modeled as a generic stochastic process. However, the integration of a priori knowledge about underlying system structures can provide insights on otherwise unobservable intrinsic states that yield the superior prediction performance and increased generalization capabilities. Methods: We present a novel method for modeling human motor behavior that explicitly includes a neuroscientifically supported model of human motor control, in which the dynamics of the human arm are modeled by a mechanical impedance that tracks a latent desired trajectory. We adopt a Bayesian setting by defining Gaussian process (GP) priors for the impedance elements and the latent desired trajectory. This enables exploitation of a priori human arm impedance knowledge for regression of interaction forces through inference of a latent desired human trajectory. Results: The method is validated using simulated data, with particular focus on effects of GP prior parameterization and intention estimation capabilities. The superior prediction performance is shown with respect to a naive GP prior. An experiment with human participants evaluates generalization capabilities and effects of training data sparsity. Conclusion: We derive the correlations of an impedance-based GP model of human motor behavior that exploits a priori knowledge. Significance: The model effectively predicts interaction forces by inferring a latent desired human trajectory in previously observed as well as unobserved regions of the input space.

While convolutional neural networks (CNNs) have been successfully applied to many challenging classification applications, they typically require large datasets for training. When the availability of labeled data is limited, data augmentation is a critical preprocessing step for CNNs. However, data augmentation for wearable sensor data has not been deeply investigated yet. In this paper, various data augmentation methods for wearable sensor data are proposed. The proposed methods and CNNs are applied to the classification of the motor state of Parkinson's Disease patients, which is challenging due to small dataset size, noisy labels, and large intra-class variability. Appropriate augmentation improves the classification performance from 77.54\% to 86.88\%.Comment: ICMI2017 (oral session

Assistive and rehabilitation devices are a promising and challenging field of recent robotics research. Motivated by societal needs such as aging populations, such devices can support motor functionality and subject training. The design, control, sensing and assessment of the devices become more sophisticated due to a human in the loop. This paper gives a human-robot interaction perspective on current issues and opportunities in the field. On the topic of control and machine learning, approaches that support but do not distract subjects are reviewed. Options to provide sensory user-feedback that are currently missing from robotic devices are outlined. Parallels between device acceptance and affective computing are made. Furthermore, requirements for functional assessment protocols that relate to real-world tasks are discussed. In all topic areas, the design of human-oriented frameworks and methods is dominated by challenges related to the close interaction between the human and robotic device. This paper discusses the aforementioned aspects in order to open up new perspectives for future robotic solutions.