Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Authors:
Top Cited Papers
- 12 July 2016
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 27 (10) , 2134-2144
Abstract
Model-free control is an important and promising topic in control fields, which has attracted extensive attention in the past few years. In this paper, we aim to solve the model-free optimal tracking control problem of nonaffine nonlinear discrete-time systems. A critic-only Q-learning (CoQL) method is developed, which learns the optimal tracking control from real system data, and thus avoids solving the tracking Hamilton-Jacobi-Bellman equation. First, the Q-learning algorithm is proposed based on the augmented system, and its convergence is established. Using only one neural network for approximating the Q-function, the CoQL method is developed to implement the Q-learning algorithm. Furthermore, the convergence of the CoQL method is proved with the consideration of neural network approximation error. With the convergent Q-function obtained from the CoQL method, the adaptive optimal tracking control is designed based on the gradient descent scheme. Finally, the effectiveness of the developed CoQL method is demonstrated through simulation studies. The developed CoQL method learns with off-policy data and implements with a critic-only structure, thus it is easy to realize and overcome the inadequate exploration problem.Keywords
Funding Information
- National Natural Science Foundation of China (61233001, 61273140, 61304086, 61374105, 61503377, 61533017, U1501251)
- Early Career Development Award of the State Key Laboratory of Management and Control for Complex Systems
- National Priorities Research Program through the Qatar National Research Fund (a member of Qatar Foundation) (NPRP 7-1482-1-278)
This publication has 72 references indexed in Scilit:
- Numerical adaptive learning control scheme for discrete‐time non‐linear systemsIET Control Theory & Applications, 2013
- Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systemsAutomatica, 2012
- Neural Network Based Online Simultaneous Policy Update Algorithm for Solving the HJI Equation in Nonlinear $H_{\infty}$ ControlIEEE Transactions on Neural Networks and Learning Systems, 2012
- Optimal tracking and two-channel disturbance rejection under control energy constraintAutomatica, 2011
- Model-free
H
∞
control design for unknown linear discrete-time systems via Q-learning with LMIAutomatica, 2010
- New algorithms of the Q-learning typeAutomatica, 2008
- Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity controlAutomatica, 2007
- A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systemsNeural Networks, 2006
- Asynchronous stochastic approximation and Q-learningMachine Learning, 1994
- Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networksNeural Networks, 1990