Black-box data-efficient policy search for robotics
- 1 September 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 51-58
Abstract
The most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynamical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot).Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Emergency response to the nuclear accident at the Fukushima Daiichi Nuclear Power Plants using mobile rescue robotsJournal of Field Robotics, 2012
- Model learning for robot control: a surveyCognitive Processing, 2011
- A Survey on Policy Search for RoboticsFoundations and Trends in Robotics, 2011
- Policy search for motor primitives in roboticsMachine Learning, 2010
- Policy Gradients with Parameter-Based Exploration for Control2008
- Reinforcement learning of motor skills with policy gradientsNeural Networks, 2008
- Evolutionary Optimization in Uncertain Environments—A SurveyIEEE Transactions on Evolutionary Computation, 2005
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001
- Genetic Algorithms, Selection Schemes, and the Varying Effects of NoiseEvolutionary Computation, 1996
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992