Hyperbolically Discounted Temporal Difference Learning
Authors:
- William H. AlexanderWAWilliam H. Alexander
- Joshua W. BrownJBJoshua W. Brown
- 1 June 2010
- journal article
- Published by MIT Press in Neural Computation
- Vol. 22 (6) , 1511-1527
Abstract
Hyperbolic discounting of future outcomes is widely observed to underlie choice behavior in animals. Additionally, recent studies (Kobayashi & Schultz, 2008 ) have reported that hyperbolic discounting is observed even in neural systems underlying choice. However, the most prevalent models of temporal discounting, such as temporal difference learning, assume that future outcomes are discounted exponentially. Exponential discounting has been preferred largely because it can be expressed recursively, whereas hyperbolic discounting has heretofore been thought not to have a recursive definition. In this letter, we define a learning algorithm, hyperbolically discounted temporal difference (HDTD) learning, which constitutes a recursive formulation of the hyperbolic model.Keywords
This publication has 16 references indexed in Scilit:
- Influence of Reward Delays on Responses of Dopamine NeuronsThe Journal of Neuroscience, 2008
- Long-Term Reward Prediction in TD Models of the Dopamine SystemNeural Computation, 2002
- Reward-Predicting and Reward-Detecting Neuronal Activity in the Primate Supplementary Eye FieldJournal of Neurophysiology, 2000
- Behavioral considerations suggest an average reward TD model of the dopamine systemNeurocomputing, 2000
- Preference for sequences of rewards: further tests of a parallel discounting modelBehavioural Processes, 1999
- Risky Theories—The Effects of Variance on Foraging DecisionsAmerican Zoologist, 1996
- Exponential Versus Hyperbolic Discounting of Delayed Outcomes: Risk and Waiting TimeAmerican Zoologist, 1996
- DISCOUNTING OF DELAYED REWARDS: MODELS OF INDIVIDUAL CHOICEJournal of the Experimental Analysis of Behavior, 1995
- Chapter 15 Reward-related activity in the monkey striatum and substantia nigra1993
- Neuronal activity in monkey ventral striatum related to the expectation of rewardJournal of Neuroscience, 1992