Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
Preprint
Abstract
Policy gradient methods are a powerful family of reinforcement learning algorithms for continuous control that optimize a policy directly. However, standard first-order methods often converge slowly. Second-order methods can accelerate learning by using curvature information, but they are typically expensive to compute. The linear quadratic regulator (LQR) is a practical setting in which key quantities, such as the policy gradient, admit closed-form expressions. In this work, we develop second-order policy gradient algorithms for LQR by deriving explicit formulas for both the approximate and exact Hessians used in Gauss--Newton and Newton methods, respectively. Numerical experiments show a faster convergence rate for the proposed second-order approach over the standard first-order policy gradient baseline.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Convergence and Robustness of Value and Policy Iteration for the Linear Quadratic Regulator2025
- Global Optimality Guarantees for Policy Gradient MethodsOperations Research, 2024
- Quasi-Newton Iteration in Deterministic Policy Gradient2022
- Optimal network-level traffic signal control: A benders decomposition-based solution algorithmTransportation Research Part B: Methodological, 2019
- A Family of Iterative Gauss-Newton Shooting Methods for Nonlinear Optimal Control2018
- Natural Gradient Works Efficiently in LearningNeural Computation, 1998
- An Economic Interpretation of Optimal Control Theory1997
- The Commutation Matrix: Some Properties and ApplicationsThe Annals of Statistics, 1979
- Applications of optimal control theory in economicsSynthese, 1975