Adaptive Control of a Mechatronic System Using Constrained Residual Reinforcement Learning
- 25 January 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Electronics
- Vol. 69 (10) , 10447-10456
Abstract
In this article, we propose a simple, practical, and intuitive approach to improve the performance of a conventional controller in uncertain environments using deep reinforcement learning while maintaining safe operation. Our approach is motivated by the observation that conventional controllers in industrial motion control value robustness over adaptivity to deal with different operating conditions and are suboptimal as a consequence. Reinforcement learning, on the other hand, can optimize a control signal directly from input–output data and thus adapts to operational conditions but lacks safety guarantees, impeding its use in industrial environments. To realize adaptive control using reinforcement learning in such conditions, we follow a residual learning methodology, where a reinforcement learning algorithm learns corrective adaptations to a base controller’s output to increase optimality. We investigate how constraining the residual agent’s actions enables to leverage the base controller’s robustness to guarantee safe operation. We detail the algorithmic design and propose to constrain the residual actions relative to the base controller to increase the method’s robustness. Building on Lyapunov stability theory, we prove stability for a broad class of mechatronic closed-loop systems. We validate our method experimentally on a slider crank setup and investigate how the constraints affect the safety during learning and optimality after convergence.Keywords
All Related Versions
Funding Information
- Vlaamse regering
This publication has 27 references indexed in Scilit:
- Stochastic Model Predictive Control: An Overview and Perspectives for Future ResearchIEEE Control Systems, 2016
- Tube-Based Robust Model Predictive Control of Nonlinear Systems via Collective Neurodynamic OptimizationIEEE Transactions on Industrial Electronics, 2016
- Modeling and simulation of flexible slider-crank mechanism with clearance for a closed high speed press systemMechanism and Machine Theory, 2014
- System identification of nonlinear state-space modelsAutomatica, 2011
- Robust adaptive MPC for constrained uncertain nonlinear systemsInternational Journal of Adaptive Control and Signal Processing, 2010
- Robust tube-based MPC for tracking of constrained linear systems with additive disturbancesJournal of Process Control, 2010
- Survey on iterative learning control, repetitive control, and run-to-run controlJournal of Process Control, 2009
- Constrained model predictive control: Stability and optimalityAutomatica, 2000
- An adaptive PID learning control of robot manipulatorsAutomatica, 2000
- The role and use of the stochastic linear-quadratic-Gaussian problem in control system designIEEE Transactions on Automatic Control, 1971