All-Action Policy Gradient Methods: A Numerical Integration Approach
Preprint
Abstract
While often stated as an instance of the likelihood ratio trick [Rubinstein, 1989], the original policy gradient theorem [Sutton, 1999] involves an integral over the action space. When this integral can be computed, the resulting "all-action" estimator [Sutton, 2001] provides a conditioning effect [Bratley, 1987] reducing the variance significantly compared to the REINFORCE estimator [Williams, 1992]. In this paper, we adopt a numerical integration perspective to broaden the applicability of the all-action estimator to general spaces and to any function class for the policy or critic components, beyond the Gaussian case considered by [Ciosek, 2018]. In addition, we provide a new theoretical result on the effect of using a biased critic which offers more guidance than the previous "compatible features" condition of [Sutton, 1999]. We demonstrate the benefit of our approach in continuous control tasks with nonlinear function approximation. Our results show improved performance and sample efficiency.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Infinite-Horizon Policy-Gradient EstimationJournal of Artificial Intelligence Research, 2001
- Markov Decision Processes: Discrete Stochastic Dynamic ProgrammingTechnometrics, 1995
- Advantage Updating1993
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning1992
- Sensitivity Analysis for Simulations via Likelihood RatiosOperations Research, 1989
- Sensitivity Analysis and Performance Extrapolation for Computer Simulation ModelsOperations Research, 1989
- Simulation methods for queues: An overviewQueueing Systems, 1988