ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning
Authors:
- 23 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7505-7512
Abstract
Building assistive interfaces for controlling robots through arbitrary, high-dimensional, noisy inputs (e.g., webcam images of eye gaze) can be challenging, especially when it involves inferring the user's desired action in the absence of a natural ‘default’ interface. Reinforcement learning from online user feedback on the system's performance presents a natural solution to this problem, and enables the interface to adapt to individual users. However, this approach tends to require a large amount of human-in-the-loop training data, especially when feedback is sparse. We propose a hierarchical solution that learns efficiently from sparse user feedback: we use offline pre-training to acquire a latent embedding space of useful, high-level robot behaviors, which, in turn, enables the system to focus on using online user feedback to learn a mapping from user inputs to desired high-level behaviors. The key insight is that access to a pre-trained policy enables the system to learn more from sparse rewards than a naïve RL algorithm: using the pre-trained policy, the system can make use of successful task executions to relabel, in hindsight, what the user actually meant to do during unsuccessful executions. We evaluate our method primarily through a user study with 12 participants who perform tasks in three simulated robotic manipulation domains using a webcam and their eye gaze: flipping light switches, opening a shelf door to reach objects inside, and rotating a valve. The results show that our method successfully learns to map 128-dimensional gaze features to 7-dimensional joint torques from sparse rewards in under 10 minutes of online training, and seamlessly helps users who employ different gaze strategies, while adapting to distributional shift in webcam inputs, tasks, and environmentsKeywords
All Related Versions
Funding Information
- NSF (IIS-1651843)
This publication has 36 references indexed in Scilit:
- Anticipating Human Activities Using Object Affordances for Reactive Robotic ResponseIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Continuous Closed-Loop Decoder Adaptation with a Recursive Maximum Likelihood Algorithm Allows for Rapid Performance Acquisition in Brain-Machine InterfacesNeural Computation, 2014
- Recognition, prediction, and planning for assisted teleoperation of freeform tasksAutonomous Robots, 2013
- A high-performance neural prosthesis enabled by control algorithm designNature Neuroscience, 2012
- User-Centered Design of a Dynamic-Autonomy Remote Interaction Concept for Manipulation-Capable Robots to Assist Elderly People in the HomeJournal of Human-Robot Interaction, 2012
- A closed-loop human simulator for investigating the role of feedback control in brain-machine interfacesJournal of Neurophysiology, 2011
- A Wireless Brain-Machine Interface for Real-Time Speech SynthesisPLOS ONE, 2009
- Continuous Shared Control for Stabilizing Reaching and Grasping With Brain-Machine InterfacesIEEE Transactions on Biomedical Engineering, 2006
- Integration of a Rehabilitation Robotic System (KARES II) with Human-Friendly Man-Machine Interaction UnitsAutonomous Robots, 2004
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998