Reinforcement Q-Learning Algorithm for H∞ Tracking Control of Unknown Discrete-Time Linear Systems
- 20 December 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 50 (11) , 4109-4122
Abstract
This article addresses the online reinforcement Q-learning algorithms to design H∞ tracking controller for unknown discrete-time linear systems. An augmented system composed of the original system and the command generator is constructed, and a discounted performance function is introduced to establish a discounted game algebraic Riccati equation (GARE). The existence conditions of a solution to the GARE are proposed and a lower bound is found for the discount factor to assure the stability of the H∞ tracking control solution. The Q-function Bellman equation is then derived, based on which the reinforcement Q-learning algorithm is developed to learn the solution to H∞ tracking control problem without knowing the system dynamics. Both state-data-driven and output-data-driven reinforcement Q-learning algorithms toward finding the control policies are proposed. Unlike the value function approximation (VFA)-based approach, it is proved that the Q-learning scheme brings out no bias of solution to the Q-function Bellman equation under the probing noise satisfying the persistent excitation (PE) condition, and therefore, converges to the nominal discounted GARE solution. Moreover, the proposed output-data-driven method is more powerful than the state-data-driven method as it may not be available to completely measure the full system states in practical applications. A simulation example with a single-phase voltage-source UPS inverter is used to verify the effectiveness of the proposed Q-learning algorithms.Keywords
Funding Information
- National Natural Science Foundation (61573154)
- Science and Technology Planning Project of Guangdong Province (2015A010106003, 2017A010101009)
This publication has 44 references indexed in Scilit:
- Linear Quadratic Tracking Control of Partially-Unknown Continuous-Time Systems Using Reinforcement LearningIEEE Transactions on Automatic Control, 2014
- Integral Reinforcement Learning for Linear Continuous-Time Zero-Sum Games With Completely Unknown DynamicsIEEE Transactions on Automation Science and Engineering, 2014
- Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear SystemsIEEE Transactions on Neural Networks and Learning Systems, 2013
- Dynamic Learning From Adaptive Neural Network Control of a Class of Nonaffine Nonlinear SystemsIEEE Transactions on Neural Networks and Learning Systems, 2013
- Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamicsAutomatica, 2012
- Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton–Jacobi equationsAutomatica, 2011
- Adaptive optimal control for continuous-time linear systems based on policy iterationAutomatica, 2009
- Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity controlAutomatica, 2007
- The discrete-time Riccati equation related to the H/sub ∞/ control problemIEEE Transactions on Automatic Control, 1994
- An iterative technique for the computation of the steady state gains for the discrete optimal regulatorIEEE Transactions on Automatic Control, 1971