Learning From Guided Play: Improving Exploration for Adversarial Imitation Learning With Simple Auxiliary Tasks
- 13 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (3) , 1263-1270
Abstract
Adversarial imitation learning (AIL) has become a popular alternative to supervised imitation learning that reduces the distribution shift suffered by the latter. However, AIL requires effective exploration during an online reinforcement learning phase. In this work, we show that the standard, naïve approach to exploration can manifest as a suboptimal local maximum if a policy learned with AIL sufficiently matches the expert distribution without fully learning the desired task. This can be particularly catastrophic for manipulation tasks, where the difference between an expert and a non-expert state-action pair is often subtle. We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of multiple exploratory, auxiliary tasks in addition to a main task. The addition of these auxiliary tasks forces the agent to explore states and actions that standard AIL may learn to ignore. Additionally, this particular formulation allows for the reusability of expert data between main tasks. Our experimental results in a challenging multitask robotic manipulation domain indicate that LfGP significantly outperforms both AIL and behaviour cloning, while also being more expert sample efficient than these baselines. To explain this performance gap, we provide further analysis of a toy problem that highlights the coupling between a local maximum and poor exploration, and also visualize the differences between the learned models from AIL and LfGP.Keywords
All Related Versions
Funding Information
- Canada Research Chairs
This publication has 10 references indexed in Scilit:
- Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations2021
- Hierarchical Reinforcement LearningACM Computing Surveys, 2021
- Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention2021
- First return, then exploreNature, 2021
- End-to-End Driving Via Conditional Imitation Learning2018
- Overcoming Exploration in Reinforcement Learning with Demonstrations2018
- OptionGAN: Learning Joint Reward-Policy Options Using Generative Adversarial Inverse Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Apprenticeship learning via inverse reinforcement learning2004
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998