A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Open Access
- 27 December 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 36 (2) , 328-347
Abstract
Most policy search (PS) algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data,” we refer to this challenge as “micro-data reinforcement learning.” In this article, we show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based PS), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots, designing generic priors, and optimizing the computing time.Keywords
All Related Versions
Funding Information
- European Research Council (637972)
- European Commission (731540)
- MEMMO (780684)
- European Unions Horizon 2020 Research and Innovation Programme (739578)
This publication has 130 references indexed in Scilit:
- Reinforcement learning in robotics: A surveyThe International Journal of Robotics Research, 2013
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Central pattern generators for locomotion control in animals and robots: A reviewNeural Networks, 2008
- Resilient Machines Through Continuous Self-ModelingScience, 2006
- Evolving Neural Networks through Augmenting TopologiesEvolutionary Computation, 2002
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Genetic Algorithms, Selection Schemes, and the Varying Effects of NoiseEvolutionary Computation, 1996
- Iterative learning control: A survey and new resultsJournal of Robotic Systems, 1992
- A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of NoiseJournal of Basic Engineering, 1964
- A New Approach to Linear Filtering and Prediction ProblemsJournal of Basic Engineering, 1960