Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient
- 6 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (2) , 1031-1048
Abstract
By introducing randomness on the environments, domain randomization (DR) imposes diversity to the policy training of deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance incurred by trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing a low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. Based on our theory, we further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity for the practical implementation. By dividing the entire space of environments into some subspaces and then estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of variance reduction and faster convergence than the state-dependent baselines. Empirical evaluations on six robot control tasks with randomized dynamics demonstrate that VRDR not only accelerates the convergence of policy training, but can consistently achieve a better eventual policy with improved training stability.Keywords
Funding Information
- National Natural Science Foundation of China (62250055, T2122024, 62125109, 61831018, 62371288, 62320106003, 61931023, 61932022, 61972256, 61971285, 62120106007)
- Program of Shanghai Science and Technology Innovation (20511100100)
- Key Technology Research and Development Program of Shandong Province (2020CXGC010701)
This publication has 20 references indexed in Scilit:
- Data-Efficient Domain Randomization With Bayesian OptimizationIEEE Robotics and Automation Letters, 2021
- Accelerated Variance Reduction Stochastic ADMM for Large-Scale Machine LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Mastering Complex Control in MOBA Games with Deep Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Learning dexterous in-hand manipulationThe International Journal of Robotics Research, 2019
- Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight2019
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization2018
- Domain randomization for transferring deep neural networks from simulation to the real world2017
- Artificial Intelligence with Uncertainty2017
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic ProgrammingSIAM Journal on Optimization, 2013
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010