Quasi-Newton Iteration in Deterministic Policy Gradient
Authors:
- 8 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2124-2129
Abstract
This paper presents a model-free approximation for the Hessian of the performance of deterministic policies to use in the context of Reinforcement Learning based on Quasi-Newton steps in the policy parameters. We show that the approximate Hessian converges to the exact Hessian at the optimal policy, and allows for a superlinear convergence in the learning, provided that the policy parametrization is rich. The natural policy gradient method can be interpreted as a particular case of the proposed method. We analytically verify the formulation in a simple linear case and compare the convergence of the proposed method with the natural policy gradient in a nonlinear example.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Benchmarking the Natural Gradient in Policy Gradient Methods and Evolution Strategies2021
- Data-Driven Economic NMPC Using Reinforcement LearningIEEE Transactions on Automatic Control, 2019
- An Introduction to Deep Reinforcement Learning2018
- Third-Order Tensors as Linear Operators on a Space of MatricesLinear Algebra and its Applications, 2010
- Natural Actor-Critic2005
- Natural Gradient Works Efficiently in LearningNeural Computation, 1998