EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning
Authors:
- 1 November 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5041-5048
Abstract
Although imitation learning is often used in robotics, the approach frequently suffers from data mismatch and compounding errors. DAgger is an iterative algorithm that addresses these issues by aggregating training data from both the expert and novice policies, but does not consider the impact of safety. We present a probabilistic extension to DAgger, which attempts to quantity the confidence of the novice policy as a proxy for safety. Our method, EnsembleDAgger, approximates a Gaussian Process using an ensemble of neural networks. Using the variance as a measure of confidence, we compute a decision rule that captures how much we doubt the novice, thus determining when it is safe to allow the novice to act. With this approach, we aim to maximize the novice’s share of actions, while constraining the probability of failure. We demonstrate improved safety and learning performance compared to other DAgger variants and classic imitation learning on an inverted pendulum and in the MuJoCo HalfCheetah environment.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Safe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes2016
- SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces2016
- Reachability-based safe learning with Gaussian processes2014
- Maximum Mean Discrepancy Imitation Learning2013
- Imitation and Reinforcement LearningIEEE Robotics & Automation Magazine, 2010
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Search-based structured predictionMachine Learning, 2009
- Gaussian Processes for Machine Learning2005
- Accelerating Reinforcement Learning through Implicit ImitationJournal of Artificial Intelligence Research, 2003
- Ensembling neural networks: Many could be better than allArtificial Intelligence, 2002