An Online Policy Gradient Algorithm for Markov Decision Processes with Continuous States and Actions
- 1 March 2016
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 28 (3) , 563-593
Abstract
We consider the learning problem under an online Markov decision process (MDP) aimed at learning the time-dependent decision-making policy of an agent that minimizes the regret—the difference from the best fixed policy. The difficulty of online MDP learning is that the reward function changes over time. In this letter, we show that a simple online policy gradient algorithm achieves regret for T steps under a certain concavity assumption and under a strong concavity assumption. To the best of our knowledge, this is the first work to present an online MDP algorithm that can handle continuous state, action, and parameter spaces with guarantee. We also illustrate the behavior of the proposed online policy gradient method through experiments.Keywords
This publication has 9 references indexed in Scilit:
- An Online Policy Gradient Algorithm for Markov Decision Processes with Continuous States and Actions2014
- Online Markov Decision Processes Under Bandit FeedbackIEEE Transactions on Automatic Control, 2013
- Parameter-exploring policy gradientsNeural Networks, 2010
- Online Markov Decision ProcessesMathematics of Operations Research, 2009
- Markov Decision Processes with Arbitrary Reward ProcessesMathematics of Operations Research, 2009
- Logarithmic regret algorithms for online convex optimizationMachine Learning, 2007
- Policy Gradient Methods for Robotics2006
- Efficient algorithms for online decision problemsJournal of Computer and System Sciences, 2005
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992