An Online Prediction Algorithm for Reinforcement Learning with Linear Function Approximation using Cross Entropy Method
Authors:
- Ajin George JosephAJAjin George Joseph
- Shalabh BhatnagarSBShalabh Bhatnagar
Preprint
Abstract
In this paper, we provide two new stable online algorithms for the problem of prediction in reinforcement learning, \emph{i.e.}, estimating the value function of a model-free Markov reward process using the linear function approximation architecture and with memory and computation costs scaling quadratically in the size of the feature set. The algorithms employ the multi-timescale stochastic approximation variant of the very popular cross entropy (CE) optimization method which is a model based search method to find the global optimum of a real-valued function. A proof of convergence of the algorithms using the ODE method is provided. We supplement our theoretical results with experimental comparisons. The algorithms achieve good performance fairly consistently on many RL benchmark problems with regards to computational efficiency, accuracy and stability.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Basis Function Adaptation in Temporal Difference Reinforcement LearningAnnals of Operations Research, 2005
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005
- Model-Based Search for Combinatorial Optimization: A Critical SurveyAnnals of Operations Research, 2004
- Least Squares Policy Evaluation Algorithms with Linear Function ApproximationDiscrete Event Dynamic Systems, 2003
- Technical Update: Least-Squares Temporal Difference LearningMachine Learning, 2002
- An analysis of temporal-difference learning with function approximationIEEE Transactions on Automatic Control, 1997
- Ant colony system: a cooperative learning approach to the traveling salesman problemIEEE Transactions on Evolutionary Computation, 1997
- Stochastic approximation with two time scalesSystems & Control Letters, 1997
- Adaptive Algorithms and Stochastic Approximations1990
- Stochastic Approximation Methods for Constrained and Unconstrained Systems1978