StROL: Stabilized and Robust Online Learning From Humans
- 16 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (3) , 2303-2310
Abstract
Robots often need to learn the human's reward function online, during the current interaction. This real-time learning requires fast but approximate learning rules: when the human's behavior is noisy or suboptimal, current approximations can result in unstable robot learning. Accordingly, in this letter we seek to enhance the robustness and convergence properties of gradient descent learning rules when inferring the human's reward parameters. We model the robot's learning algorithm as a dynamical system over the human preference parameters, where the human's true (but unknown) preferences are the equilibrium point. This enables us to perform Lyapunov stability analysis to derive the conditions under which the robot's learning dynamics converge. Our proposed algorithm (StROL) uses these conditions to learn robust-by-design learning rules: given the original learning dynamics, StROL outputs a modified learning rule that now converges to the human's true parameters under a larger set of human inputs. In practice, these autonomously generated learning rules can correctly infer what the human is trying to convey, even when the human is noisy, biased, and suboptimal. Across simulations and a user study we find that StROL results in a more accurate estimate and less regret than state-of-the-art approaches for online reward learning.Keywords
All Related Versions
Funding Information
- NSF (#2205241)
This publication has 23 references indexed in Scilit:
- Learning From Human Directional CorrectionsIEEE Transactions on Robotics, 2022
- Expert Intervention LearningAutonomous Robots, 2021
- Analyzing Human Models that Adapt Online2021
- Corrective Shared Autonomy for Addressing Task VariabilityIEEE Robotics and Automation Letters, 2021
- Learning Occupancy Priors of Human Motion From Semantic Maps of Urban EnvironmentsIEEE Robotics and Automation Letters, 2021
- Preference-Based Learning for Exoskeleton Gait Optimization2020
- Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical CorrectionsIEEE Transactions on Robotics, 2020
- Differential game theory for versatile physical human–robot interactionNature Machine Intelligence, 2019
- A dynamical system approach to task-adaptation in physical human–robot interactionAutonomous Robots, 2018
- Rational Use of Cognitive Resources: Levels of Analysis Between the Computational and the AlgorithmicTopics in Cognitive Science, 2015