Roll-Drop: accounting for observation noise with a single parameter
Authors:
- Daniele De MartiniDMDaniele De Martini
- Siddhant GangapurwalaSGSiddhant Gangapurwala
- Wolfgang MerktWMWolfgang Merkt
- Ioannis HavoutisIHIoannis Havoutis
Preprint
Abstract
This paper proposes a simple strategy for sim-to-real in Deep-Reinforcement Learning (DRL) -- called Roll-Drop -- that uses dropout during simulation to account for observation noise during deployment without explicitly modelling its distribution for each state. DRL is a promising approach to control robots for highly dynamic and feedback-based manoeuvres, and accurate simulators are crucial to providing cheap and abundant data to learn the desired behaviour. Nevertheless, the simulated data are noiseless and generally show a distributional shift that challenges the deployment on real machines where sensor readings are affected by noise. The standard solution is modelling the latter and injecting it during training; while this requires a thorough system identification, Roll-Drop enhances the robustness to sensor noise by tuning only a single parameter. We demonstrate an 80% success rate when up to 25% noise is injected in the observations, with twice higher robustness than the baselines. We deploy the controller trained in simulation on a Unitree A1 platform and assess this improved robustness on the physical system.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- RLOC: Terrain-Aware Legged Locomotion Using Reinforcement Learning and Optimal ControlIEEE Transactions on Robotics, 2022
- Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged LocomotionIEEE Robotics and Automation Letters, 2022
- Learning robust perceptive locomotion for quadrupedal robots in the wildScience Robotics, 2022
- An Efficient Locally Reactive Controller for Safe Navigation in Visual Teach and Repeat MissionsIEEE Robotics and Automation Letters, 2022
- Multi-expert learning of adaptive legged locomotionScience Robotics, 2020
- Crossing the Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics2020
- Learning quadrupedal locomotion over challenging terrainScience Robotics, 2020
- Learning agile and dynamic motor skills for legged robotsScience Robotics, 2019
- Domain randomization for transferring deep neural networks from simulation to the real world2017
- ANYmal - a highly mobile and dynamic quadrupedal robot2016