Reinforcement learning with state-dependent discount factor
Authors:
- 1 August 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Conventional reinforcement learning algorithms have several parameters which determine the feature of learning process, called meta-parameters. In this study, we focus on the discount factor that influences the time scale of the tradeoff between immediate and delayed rewards. The discount factor is usually considered as a constant value, but we introduce the state-dependent discount function and a new optimization criterion for the reinforcement learning algorithm. We first derive a new algorithm under the criterion, named ExQ-learning and we prove that the algorithm converges to the optimal action-value function in the meaning of new criterion w.p.1. We then present a framework to optimize the discount factor and the discount function by using an evolutionary algorithm. In order to validate the proposed method, we conduct a simple computer simulation and show that the proposed algorithm can find an appropriate state-dependent discount function with which performs better than that with a constant discount factor.Keywords
This publication has 11 references indexed in Scilit:
- Tuning-free step-size adaptation2012
- The Role of Serotonin in the Regulation of Patience and ImpulsivityMolecular Neurobiology, 2012
- Intrinsically Motivated Reinforcement Learning: An Evolutionary PerspectiveIEEE Transactions on Autonomous Mental Development, 2010
- Control of exploitation–exploration meta-parameter in reinforcement learningNeural Networks, 2002
- Metalearning and neuromodulationNeural Networks, 2002
- A Neural Substrate of Prediction and RewardScience, 1997
- Average reward reinforcement learning: Foundations, algorithms, and empirical resultsMachine Learning, 1996
- On the Convergence of Stochastic Iterative Dynamic Programming AlgorithmsNeural Computation, 1994
- Optimal foraging of a reproducing animal as a discounted reward problemJournal of Applied Probability, 1994
- Q-learningMachine Learning, 1992