Benjamin Petit
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Benjamin Petit has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Jimmy T. H. Smith
Stanford University Stanford CA
1 shared publication Loren Amdahl-Culleton
Stanford University Stanford CA
1 shared publication- Pierre-Luc Bacon
Stanford University
1 shared publication - Yao Liu
School of Tourism Data, Guilin Tourism University, 26 Liangfeng Road, Yanshan District, Guilin City, Guangxi 541006, P. R. China
1 shared publication
Latest Publications
- Open Access
All-Action Policy Gradient Methods: A Numerical Integration Approach
While often stated as an instance of the likelihood ratio trick [Rubinstein, 1989], the original policy gradient theorem [Sutton, 1999] involves an integral over the action space. When this integral can be computed, the resulting "all-action" estimator [Sutton, 2001] provides a conditioning effect [Bratley, 1987] reducing the variance significantly compared to the REINFORCE estimator [Williams, 1992]. In this paper, we adopt a numerical integration perspective to broaden the applicability of the all-action estimator to general spaces and to any function class for the policy or critic components, beyond the Gaussian case considered by [Ciosek, 2018]. In addition, we provide a new theoretical result on the effect of using a biased critic which offers more guidance than the previous "compatible features" condition of [Sutton, 1999]. We demonstrate the benefit of our approach in continuous control tasks with nonlinear function approximation. Our results show improved performance and sample efficiency.
Open AccessAll-Action Policy Gradient Methods: A Numerical Integration Approach
While often stated as an instance of the likelihood ratio trick [Rubinstein, 1989], the original policy gradient theorem [Sutton, 1999] involves an integral over the action space. When this integral can be computed, the resulting "all-action" estimator [Sutton, 2001] provides a conditioning effect [Bratley, 1987] reducing the variance significantly compared to the REINFORCE estimator [Williams, 1992]. In this paper, we adopt a numerical integration perspective to broaden the applicability of the all-action estimator to general spaces and to any function class for the policy or critic components, beyond the Gaussian case considered by [Ciosek, 2018]. In addition, we provide a new theoretical result on the effect of using a biased critic which offers more guidance than the previous "compatible features" condition of [Sutton, 1999]. We demonstrate the benefit of our approach in continuous control tasks with nonlinear function approximation. Our results show improved performance and sample efficiency.