Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention
Authors:
- 30 May 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6664-6671
Abstract
Reinforcement Learning (RL) algorithms can in principle acquire complex robotic skills by learning from large amounts of data in the real world, collected via trial and error. However, most RL algorithms use a carefully engineered setup in order to collect data, requiring human supervision and intervention to provide episodic resets. This is particularly evident in challenging robotics problems, such as dexterous manipulation. To make data collection scalable, such applications require reset-free algorithms that are able to learn autonomously, without explicit instrumentation or human intervention. Most prior work in this area handles single-task learning. However, we might also want robots that can perform large repertoires of skills. At first, this would appear to only make the problem harder. However, the key observation we make in this work is that an appropriately chosen multi-task RL setting actually alleviates the reset-free learning challenge, with minimal additional machinery required. In effect, solving a multi-task problem can directly solve the reset-free problem since different combinations of tasks can serve to perform resets for other tasks. By learning multiple tasks together and appropriately sequencing them, we can effectively learn all of the tasks together reset-free. This type of multi-task learning can effectively scale reset-free learning schemes to much more complex problems, as we demonstrate in our experiments. We propose a simple scheme for multi-task learning that tackles the reset-free learning problem, and show its effectiveness at learning to solve complex dexterous manipulation tasks in both hardware and simulation without any explicit resets. This work shows the ability to learn in-hand manipulation behaviors in the real world with RL without any human intervention.Keywords
All Related Versions
Funding Information
- Office of Naval Research
- National Science Foundation
This publication has 21 references indexed in Scilit:
- Optimal control with learned local models: Application to dexterous manipulation2016
- Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours2016
- Learning robot in-hand manipulation with tactile features2015
- Learning compound multi-step controllers under unknown dynamics2015
- Bayesian optimization for learning gaits under uncertaintyAnnals of Mathematics and Artificial Intelligence, 2015
- Learning contact-rich manipulation skills with guided policy search2015
- Multi-task policy search for robotics2014
- Real-time behaviour synthesis for dynamic hand-manipulation2014
- Learning to grasp everyday objects using reinforcement-learning with automatic value cut-off2007
- Synthesizing animations of human manipulation tasks2004