Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots
Preprint
Abstract
Learning for control can acquire controllers for novel robotic tasks, paving the path for autonomous agents. Such controllers can be expert-designed policies, which typically require tuning of parameters for each task scenario. In this context, Bayesian optimization (BO) has emerged as a promising approach for automatically tuning controllers. However, when performing BO on hardware for high-dimensional policies, sample-efficiency can be an issue. Here, we develop an approach that utilizes simulation to map the original parameter space into a domain-informed space. During BO, similarity between controllers is now calculated in this transformed space. Experiments on the ATRIAS robot hardware and another bipedal robot simulation show that our approach succeeds at sample-efficiently learning controllers for multiple robots. Another question arises: What if the simulation significantly differs from hardware? To answer this, we create increasingly approximate simulators and study the effect of increasing simulation-hardware mismatch on the performance of Bayesian optimization. We also compare our approach to other approaches from literature, and find it to be more reliable, especially in cases of high mismatch. Our experiments show that our approach succeeds across different controller types, bipedal robot models and simulator fidelity levels, making it applicable to a wide range of bipedal locomotion problems.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Bayesian Optimization Using Domain Knowledge on the ATRIAS Biped2018
- Terrain-adaptive locomotion skills using deep reinforcement learningACM Transactions on Graphics, 2016
- Taking the Human Out of the Loop: A Review of Bayesian OptimizationProceedings of the IEEE, 2015
- Toward Balance Recovery With Leg Prostheses Using Neuromuscular Model ControlIEEE Transactions on Biomedical Engineering, 2015
- Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robotAutonomous Robots, 2015
- Bayesian optimization for learning gaits under uncertaintyAnnals of Mathematics and Artificial Intelligence, 2015
- A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotionThe Journal of Physiology, 2015
- Robots that can adapt like animalsNature, 2015
- Optimization‐based Full Body Control for the DARPA Robotics ChallengeJournal of Field Robotics, 2015
- A Muscle-Reflex Model That Encodes Principles of Legged Mechanics Produces Human Walking Dynamics and Muscle ActivitiesIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010