Output Tracking Control Based on Adaptive Dynamic Programming With Multistep Policy Evaluation
- 28 November 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 49 (10) , 2155-2165
Abstract
In this paper, the optimal output tracking control problem of discrete-time nonlinear systems is considered. First, the augmented system is derived and the tracking control problem is converted to the regulation problem with a discounted performance index, which relies on the solution of the Bellman equation. It is known that policy iteration and value iteration are two classical algorithms for solving the Bellman equation. Through analysis of the two algorithms, it is found that policy iteration converges fast while requires an initial admissible control policy, and value iteration avoids the requirement of an initial admissible control policy but converges slowly. To achieve the tradeoff between policy iteration and value iteration, the multistep heuristic dynamic programming (MsHDP) is proposed by using multistep policy evaluation scheme. The convergence of MsHDP algorithm is proved by demonstrating that it converges to the solution of the Bellman equation. Subsequently, neural network-based actor-critic structure is developed to implement the MsHDP algorithm. The effectiveness and advantages of the developed MsHDP method are validated through comparative simulation studies.Keywords
Funding Information
- National Natural Science Foundation of China (61503377, 61533017, U1501251)
- Qatar National Research Fund (NPRP 9 166-1-031)
This publication has 57 references indexed in Scilit:
- Online Synchronous Approximate Optimal Learning Algorithm for Multi-Player Non-Zero-Sum Games With Unknown DynamicsIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2014
- Adaptive Learning in Tracking Control Based on the Dual Critic Network DesignIEEE Transactions on Neural Networks and Learning Systems, 2013
- A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systemsAutomatica, 2013
- Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systemsAutomatica, 2012
- Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamicsAutomatica, 2012
- Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problemAutomatica, 2010
- Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systemsNeural Networks, 2009
- Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approachAutomatica, 2005
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equationAutomatica, 1997