Long-Term Reward Prediction in TD Models of the Dopamine System
- 1 November 2002
- journal article
- Published by MIT Press in Neural Computation
- Vol. 14 (11) , 2567-2583
Abstract
This article addresses the relationship between long-term reward predictions and slow-timescale neural activity in temporal difference (TD) models of the dopamine system. Such models attempt to explain how the activity of dopamine (DA) neurons relates to errors in the prediction of future rewards. Previous models have been mostly restricted to short-term predictions of rewards expected during a single, somewhat artificially defined trial. Also, the models focused exclusively on the phasic pause-and-burst activity of primate DA neurons; the neurons' slower, tonic background activity was assumed to be constant. This has led to difficulty in explaining the results of neurochemical experiments that measure indications of DA release on a slow timescale, results that seem at first glance inconsistent with a reward prediction model. In this article, we investigate a TD model of DA activity modified so as to enable it to make longer-term predictions about rewards expected far in the future. We show that these predictions manifest themselves as slow changes in the baseline error signal, which we associate with tonic DA activity. Using this model, we make new predictions about the behavior of the DA system in a number of experimental situations. Some of these predictions suggest new computational explanations for previously puzzling data, such as indications from microdialysis studies of elevated DA activity triggered by aversive events.Keywords
This publication has 20 references indexed in Scilit:
- Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward eventsNeuroscience, 2000
- Extracellular dopamine dynamics in rat caudate–putamen during experimenter-delivered and intracranial self-stimulationNeuroscience, 2000
- Time, rate, and conditioning.Psychological Review, 2000
- How the Basal Ganglia Use Parallel Excitatory and Inhibitory Learning Pathways to Selectively Respond to Unexpected Rewarding CuesThe Journal of Neuroscience, 1999
- An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential pavlovian fear conditioning in the awake rabbitBehavioural Brain Research, 1999
- Serotonin-dopamine interaction and its relevance to schizophreniaAmerican Journal of Psychiatry, 1996
- Average reward reinforcement learning: Foundations, algorithms, and empirical resultsMachine Learning, 1996
- Electrical stimulation of reward sites in the ventral tegmental area increases dopamine transmission in the nucleus accumbens of the ratBehavioural Brain Research, 1993
- Responses of monkey dopamine neurons during learning of behavioral reactionsJournal of Neurophysiology, 1992
- Blocking and enhancement of fear conditioning by appetitive CSsLearning & Behavior, 1983