Model-Based Policy Iterations for Nonlinear Systems via Controlled Hamiltonian Dynamics
Open Access
- 17 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automatic Control
- Vol. 68 (5) , 2683-2698
Abstract
The infinite-horizon optimal control problem for nonlinear systems is studied. In the context of model-based, iterative learning strategies we propose an alternative definition and construction of the temporal difference error arising in Policy Iteration strategies. In such architectures the error is computed via the evolution of the Hamiltonian function (or, possibly, of its integral) along the trajectories of the closed-loop system. Herein the temporal difference error is instead obtained via two subsequent steps: first the dynamics of the underlying costate variable in the Hamiltonian system is steered by means of a (virtual) control input in such a way that the stable invariant manifold becomes externally attractive. Then, the distance-from-invariance of the manifold, induced by approximate solutions, yields a natural candidate measure for the policy evaluation step. The policy improvement phase is then performed by means of standard gradient descent methods that allows to correctly update the weights of the underlying functional approximator. The above architecture then yields an iterative (episodic) learning scheme based on a scalar, constant reward at each iteration, the value of which is insensitive to the length of the episode, as in the original spirit of Reinforcement Learning strategies for discrete-time systems. Finally, the theory is validated by means of a numerical simulation involving an automatic flight control problem.Keywords
This publication has 21 references indexed in Scilit:
- Online adaptive algorithm for optimal control with integral reinforcement learningInternational Journal of Robust and Nonlinear Control, 2013
- Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problemAutomatica, 2010
- Adaptive optimal control for continuous-time linear systems based on policy iterationAutomatica, 2009
- Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approachAutomatica, 2005
- Adaptive dynamic programmingIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2002
- L/sub 2/-gain analysis of nonlinear systems and nonlinear state-feedback H/sub infinity / controlIEEE Transactions on Automatic Control, 1992
- Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networksNeural Networks, 1990
- Design of nonlinear automatic flight control systemsAutomatica, 1977
- Suboptimal control for the nonlinear quadratic regulator problemAutomatica, 1975
- On an iterative technique for Riccati equation computationsIEEE Transactions on Automatic Control, 1968