Computational mechanisms underlying motivation to earn symbolic reinforcers
Preprint
Abstract
Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement. Significance statement: Symbolic reinforcers, like money and points, play a critical role in our lives. Like rewards, symbolic reinforcers can be motivating and can even lead to compulsive behaviors like gambling addiction. However, we lack an understanding of how symbolic reinforcement can drive fluctuations in motivation. Here we investigated the effect of symbolic reinforcers on behaviors related to motivation during a token reinforcement learning task, using a novel reinforcement learning model and data from five monkeys. Our findings suggest that the value of a task state can affect willingness to initiate a trial, speed to choose, and persistence to complete a trial. Our model makes testable predictions for within trial fluctuations of neural activity related to values of task states.Keywords
All Related Versions
- Published version: The Journal of Neuroscience, 44 (24).
This publication has 64 references indexed in Scilit:
- Limited Encoding of Effort by Dopamine Neurons in a Cost–Benefit Trade-off TaskThe Journal of Neuroscience, 2013
- Neural Systems Underlying Aversive Conditioning in Humans with Primary and Secondary ReinforcersFrontiers in Neuroscience, 2011
- States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement LearningNeuron, 2010
- Behavioral and Neural Changes after Gains and Losses of Conditioned ReinforcersJournal of Neuroscience, 2009
- TOKEN REINFORCEMENT: A REVIEW AND ANALYSISJournal of the Experimental Analysis of Behavior, 2009
- Moment-to-Moment Tracking of State Value in the AmygdalaThe Journal of Neuroscience, 2008
- A flexible software tool for temporally-precise behavioral control in MatlabJournal of Neuroscience Methods, 2008
- Modeling the Violation of Reward Maximization and Invariance in Reinforcement SchedulesPLOS Computational Biology, 2008
- Dissociation of neural regions associated with anticipatory versus consummatory phases of incentive processingPsychophysiology, 2007
- The use of tokens as rewards and tools by chimpanzees (Pan troglodytes)Animal Cognition, 2001