Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks
Preprint
Abstract
We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to $20$ spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.
Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Recurrent neural networks for stochastic control problems with delayMathematics of Control, Signals, and Systems, 2021
- Solving high-dimensional Hamilton–Jacobi–Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacePartial Differential Equations and Applications, 2021
- Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approachJournal of Computational Physics, 2020
- A derivative-free method for solving elliptic partial differential equations with deep neural networksJournal of Computational Physics, 2020
- Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential EquationsCommunications in Mathematics and Statistics, 2017
- Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equationsComputational Optimization and Applications, 2017
- Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problemAutomatica, 2010
- On reachability and minimum cost optimal controlAutomatica, 2004
- Adapted solution of a backward stochastic differential equationSystems & Control Letters, 1990
- Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulationsJournal of Computational Physics, 1988