Robust Adaptive Ensemble Adversary Reinforcement Learning
- 8 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (4) , 12562-12568
Abstract
Reinforcement learning needs to learn policies through trial and error. The unstable policies in the early stage of training make it expensive (and time-consuming) to train directly in the real environment, which may cause disastrous consequences. The popular solution is to use the simulator to train the policy and deploy it in a real environment. However, the modeling error and external disturbance between the simulation and the real environment may fail the physical deployment, resulting in the sim2real transfer problem. In this letter, we propose a novel robust adversarial reinforcement learning framework, which uses the ensemble training of multi-adversarial agents that can adaptively adjust adversaries' strength to enhance RL policy's robustness. More specifically, we take the accumulative reward as feedback and construct a PID controller to adjust the adversary's output magnitude to perform the adversarial training well. Experiments in the simulated and the real environment show that our algorithm improves the generalization ability of the policy for the modeling error and the uncertain disturbance simultaneously, outperforming the next best prior methods across all domains. The algorithm was further proven to be effective in a sim2real transfer task through the load experiment of a real racing drone, and the tracking performance is better than the PID-based flight controller.Keywords
Funding Information
- China Postdoctoral Science Foundation (BX20220071)
- Shanghai Municipality Science and Technology Major Project (2021SHZDZX0103)
- National Key R&D Program of China (2021ZD0113502)
This publication has 14 references indexed in Scilit:
- Learning quadrupedal locomotion over challenging terrainScience Robotics, 2020
- Learn-to-Recover: Retrofitting UAVs with Reinforcement Learning-Assisted Flight Control Under Cyber-Physical Attacks2020
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots2018
- A PID Controller Approach for Stochastic Optimization of Deep Networks2018
- Ensemble learning: A surveyWIREs Data Mining and Knowledge Discovery, 2018
- Domain randomization for transferring deep neural networks from simulation to the real world2017
- Control interpretations for first-order optimization methods2017
- Flying Robots2016
- Ensemble Algorithms in Reinforcement LearningIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2008
- Robust Reinforcement LearningNeural Computation, 2005