Convergence Rates of Online Critic Value Function Approximation in Native Spaces
- 20 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Control Systems Letters
- Vol. 8, 2145-2150
Abstract
This paper derives rates of convergence of online critic methods for the estimation of the value function for a class of nonlinear optimal control problems. Assuming that the underlying value function lies in reproducing kernel Hilbert space (RKHS), we derive explicit bounds on the performance of the critic in terms of the kernel functions, the number of basis functions, and the scattered location of centers used to define the RKHS. The performance of the critic is precisely measured in terms of the power function of the scattered bases, and it can be used either in an a priori evaluation of potential bases or in an a posteriori assessments of the value function error for basis enrichment or pruning. The most concise bounds in the paper describe explicitly how the critic performance depends on the placement of centers, as measured by their fill distance in a subset that contains the trajectory of the critic. To the authors’ knowledge, precise error bounds of this form are the first of their kind for online critic formulations used in optimal control problems. In addition to their general and immediate applicability to a wide range of applications, they have the potential to constitute the groundwork for more advanced “basis-adaptive” methods for nonlinear optimal control strategies, ones that address limitations due to the dimensionality of approximations.Keywords
All Related Versions
Funding Information
- NSF (CAREER CPS-1851588, CPS-2227185, S&AS-1849198, 2137159)
- Army Research Laboratory (W911QX2320001)
This publication has 18 references indexed in Scilit:
- Handbook of Reinforcement Learning and Control2021
- Reinforcement Learning for Optimal Feedback Control2018
- A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systemsAutomatica, 2013
- Robust and Adaptive Control2013
- Reinforcement Learning and Approximate Dynamic Programming for Feedback Control2012
- Optimal Control2012
- Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problemAutomatica, 2010
- Derivative reproducing properties for kernel methods in learning theoryJournal of Computational and Applied Mathematics, 2008
- Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approachAutomatica, 2005
- Error estimates and condition numbers for radial basis function interpolationAdvances in Computational Mathematics, 1995