Continuous-time reinforcement learning for robust control under worst-case uncertainty
Authors:
- 2 November 2020
- journal article
- research article
- Published by Taylor & Francis in International Journal of Systems Science
- Vol. 52 (4) , 770-784
Abstract
Reinforcement learning (RL) is an effective method to design a robust controller for unknown nonlinear systems. Uncertainty in the worst case requires a large state-action space. Hence, it is natural to use continuous-time RL methods rather than the discretisation of the spaces. In this paper, we propose a novel continuous-time RL using neural network approximation. Our method uses worst-case uncertainty to train the continuous-time RL algorithm. The backward Euler approximation is used to approximate the time derivative of the value function. Compared with the actor–critic (AC) algorithm, our method finds the robust control policy in the presence of worst-case uncertainty by taking into account the applied actions. It is shown that the AC algorithm finds the robust controller in less episodes, but its robustness is less than the results presented by our approach. The convergence of the proposed algorithm is analysed using the contraction property and differential equation techniques. The experiments show that our approach is more robust than the model-based LQR method and the well-known AC method.Keywords
Funding Information
- Secretaría de Ciencia, Humanidades, Tecnología e Innovación (CONACyT-A1-S-8216)
- CINVESTAV (SEP-CINVESTAV-62)
This publication has 25 references indexed in Scilit:
- Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approachSystems & Control Letters, 2017
- Continuous-Time Q-Learning for Infinite-Horizon Discounted Cost Linear Quadratic Regulator ProblemsIEEE Transactions on Cybernetics, 2014
- Reinforcement learning in robotics: A surveyThe International Journal of Robotics Research, 2013
- Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking NeuronsPLOS Computational Biology, 2013
- Robust LQR Controller Design for Stabilizing and Trajectory Tracking of Inverted PendulumProcedia Engineering, 2013
- Efficient Model Learning Methods for Actor–Critic ControlIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2011
- Design and Application of Suboptimal Mixed $H_{2}/H_{\infty}$ Controllers for Networked Control SystemsIEEE Transactions on Control Systems Technology, 2011
- Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problemAutomatica, 2010
- Multiagent reinforcement learning using function approximationIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2000
- Reinforcement Learning in Continuous Time and SpaceNeural Computation, 2000