Deep Q‐learning: A robust control approach
Authors:
Open Access
- 29 October 2022
- journal article
- research article
- Published by Wiley in International Journal of Robust and Nonlinear Control
- Vol. 33 (1) , 526-544
Abstract
This work aims at constructing a bridge between robust control theory and reinforcement learning. Although, reinforcement learning has shown admirable results in complex control tasks, the agent's learning behavior is opaque. Meanwhile, system theory has several tools for analyzing and controlling dynamical systems. This article places deep Q‐learning is into a control‐oriented perspective to study its learning dynamics with well‐established techniques from robust control. An uncertain linear time‐invariant model is formulated by means of the neural tangent kernel to describe learning. This novel approach allows giving conditions for stability (convergence) of the learning and enables the analysis of the agent's behavior in frequency‐domain. The control‐oriented approach makes it possible to formulate robust controllers that inject dynamical rewards as control input in the loss function to achieve better convergence properties. Three output‐feedback controllers are synthesized: gain scheduling , dynamical , and fixed‐structure controllers. Compared to traditional deep Q‐learning techniques, which involve several heuristics, setting up the learning agent with a control‐oriented tuning methodology is more transparent and has well‐established literature. The proposed approach does not use a target network and randomized replay memory. The role of the target network is overtaken by the control input, which also exploits the temporal dependency of samples (opposed to a randomized memory buffer). Numerical simulations in different OpenAI Gym environments suggest that the controlled learning can converge faster and receive higher scores (depending on the environment) compared to the benchmark double deep Q‐learning.Keywords
All Related Versions
Funding Information
- Chalmers Tekniska Högskola
This publication has 16 references indexed in Scilit:
- Constrained Deep Q-Learning Gradually Approaching Ordinary Q-LearningFrontiers in Neurorobotics, 2019
- H∞ tracking control for linear discrete‐time systems via reinforcement learningInternational Journal of Robust and Nonlinear Control, 2019
- Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning2018
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)IEEE Access, 2018
- Human-level control through deep reinforcement learningNature, 2015
- Development of Linear-Parameter-Varying Models for AircraftJournal of Guidance, Control, and Dynamics, 2004
- Robust reinforcement learning control with static and dynamic stabilityInternational Journal of Robust and Nonlinear Control, 2001
- Residual Algorithms: Reinforcement Learning with Function Approximation1995
- Beyond singular values and loop shapesJournal of Guidance, Control, and Dynamics, 1991
- Neuronlike adaptive elements that can solve difficult learning control problemsIEEE Transactions on Systems, Man, and Cybernetics, 1983