An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time
- 4 September 2013
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 24 (12) , 2088-2100
Abstract
We consider the adaptive dynamic programming technique called Dual Heuristic Programming (DHP), which is designed to learn a critic function, when using learned model functions of the environment. DHP is designed for optimizing control problems in large and continuous state spaces. We extend DHP into a new algorithm that we call Value-Gradient Learning, VGL(λ), and prove equivalence of an instance of the new algorithm to Backpropagation Through Time for Control with a greedy policy. Not only does this equivalence provide a link between these two different approaches, but it also enables our variant of DHP to have guaranteed convergence, under certain smoothness conditions and a greedy policy, when using a general smooth nonlinear function approximator for the critic. We consider several experimental scenarios including some that prove divergence of DHP under a greedy policy, which contrasts against our proven-convergent algorithm.Keywords
This publication has 19 references indexed in Scilit:
- Simple and Fast Calculation of the Second-Order Gradients for Globalized Dual Heuristic Dynamic Programming in Neural NetworksIEEE Transactions on Neural Networks and Learning Systems, 2012
- A comparison of learning speed and ability to cope without exploration between DHP and TD(0)2012
- Finite-horizon input-constrained nonlinear optimal control using single network adaptive critics2011
- Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence ProofIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2008
- Dual heuristic programming excitation neurocontrol for generators in a multimachine power systemIEEE Transactions on Industry Applications, 2003
- A direct adaptive method for faster backpropagation learning: the RPROP algorithm2002
- Convergence of critic-based training2002
- Training strategies for critic and action neural networks in dual heuristic programming method2002
- Adaptive critic designsIEEE Transactions on Neural Networks, 1997
- Neuronlike adaptive elements that can solve difficult learning control problemsIEEE Transactions on Systems, Man, and Cybernetics, 1983