Formalizing the Relationship between Hamilton-Jacobi Reachability and Reinforcement Learning
Preprint
Abstract
We unify Hamilton-Jacobi (HJ) reachability and Reinforcement Learning (RL) through a proposed running cost formulation. We prove that the resultant travel-cost value function is the unique bounded viscosity solution of a time-dependent Hamilton-Jacobi Bellman (HJB) Partial Differential Equation (PDE) with zero terminal data, whose negative sublevel set equals the strict backward-reachable tube. Using a forward reparameterization and a contraction inducing Bellman update, we show that fixed points of small-step RL value iteration converge to the viscosity solution of the forward discounted HJB. Experiments on a classical benchmark compare learned values to semi-Lagrangian HJB ground truth and quantify error.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Iterative Reachability Estimation for Safe Reinforcement Learning2023
- Reachability Analysis Using Spectrum of Koopman OperatorIEEE Control Systems Letters, 2022
- Backward Reachability for Polynomial Systems on a Finite HorizonIEEE Transactions on Automatic Control, 2021
- Algorithms for overcoming the curse of dimensionality for certain Hamilton–Jacobi equations arising in control theory and elsewhereResearch in the Mathematical Sciences, 2016
- Semi-Lagrangian Approximation Schemes for Linear and Hamilton—Jacobi Equations2013
- The Flexible, Extensible and Efficient Toolbox of Level Set MethodsJournal of Scientific Computing, 2007
- A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic gamesIEEE Transactions on Automatic Control, 2005
- On reachability and minimum cost optimal controlAutomatica, 2004
- Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations1997
- Convergence of approximation schemes for fully nonlinear second order equationsAsymptotic Analysis, 1991