Robot Fast Adaptation to Changes in Human Engagement During Simulated Dynamic Social Interaction With Active Exploration in Parameterized Reinforcement Learning
Open Access
- 5 July 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. 10 (4) , 881-893
Abstract
Dynamic uncontrolled human-robot interactions (HRI) require robots to be able to adapt to changes in the human’s behavior and intentions. Among relevant signals, non-verbal cues such as the human’s gaze can provide the robot with important information about the human’s current engagement in the task, and whether the robot should continue its current behavior or not. However, robot reinforcement learning (RL) abilities to adapt to these non-verbal cues are still underdeveloped. Here we propose an active exploration algorithm for RL during HRI where the reward function is the weighted sum of the human’s current engagement and variations of this engagement. We use a parameterized action space where a meta-learning algorithm is applied to simultaneously tune the exploration in discrete action space (e.g. moving an object) and in the space of continuous characteristics of movement (e.g. velocity, direction, strength, expressivity). We first show that this algorithm reaches state-of-the-art performance in the non-stationary multi-armed bandit paradigm. We then apply it to a simulated HRI task, and show that it outperforms continuous parameterized RL with either passive or active exploration based on different existing methods. We finally test the performance in a more realistic test of the same HRI task, where a practical approach is followed to estimate human engagement through visual cues of the head pose. The algorithm can detect and adapt to perturbations in human engagement with different durations. Altogether, these results suggest a novel efficient and robust framework for robot learning during dynamic HRI scenarios.Keywords
Funding Information
- EU-funded Project BabyRobot (687831)
- Agence Nationale de la Recherche
- Labex
- Centre National de la Recherche Scientifique
- EU-funded Project BabyRobot (687831)
This publication has 39 references indexed in Scilit:
- Active learning of inverse models with intrinsically motivated goal exploration in robotsRobotics and Autonomous Systems, 2013
- I Reach Faster When I See You Look: Gaze Effects in Human–Human and Human–Robot Face-to-Face CooperationFrontiers in Neurorobotics, 2012
- Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning ModelFrontiers in Neurorobotics, 2011
- Minds Made for Sharing: Initiating Joint Attention Recruits Reward-related NeurocircuitryJournal of Cognitive Neuroscience, 2010
- Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitationNature Neuroscience, 2009
- What is user engagement? A conceptual framework for defining user engagement with technologyJournal of the American Society for Information Science and Technology, 2008
- Cortical substrates for exploratory decisions in humansNature, 2006
- Developmental robotics, optimal artificial curiosity, creativity, music, and the fine artsConnection Science, 2006
- Explorations in engagement for humans and robotsArtificial Intelligence, 2005
- Creating conversational interfaces for childrenIEEE Transactions on Speech and Audio Processing, 2002