Reward prediction error neurons implement an efficient code for reward
Preprint
Abstract
We use efficient coding principles borrowed from sensory neuroscience to derive the optimal population of neurons to encode rewards from a probability distribution. We find that the response properties of dopaminergic reward prediction error neurons in a rodent and a primate data set are similar to those of the efficient code in many ways: the neurons have a broad distribution of midpoints covering the reward distribution; neurons with higher thresholds have higher gains, more convex tuning functions, and lower slopes; moreover, their slope is higher when the reward distribution is narrower. Furthermore, we derive learning rules that converge to this efficient code; the learning rule for the position of the neuron on the reward axis closely resembles the learning rule of distributional reinforcement learning. Thus, reward prediction error neuron responses may be optimized to broadcast an efficient reward signal, forming a connection between efficient coding and reinforcement learning, two of the most successful theories in computational neuroscience.Keywords
All Related Versions
- Published version: Nature Neuroscience, 27 (7), 1333.
This publication has 36 references indexed in Scilit:
- Efficient coding and the neural representation of valueAnnals of the New York Academy of Sciences, 2012
- Multiple Forms of Value Learning and the Function of Dopamine2009
- Influence of Reward Delays on Responses of Dopamine NeuronsThe Journal of Neuroscience, 2008
- Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewardsNature Neuroscience, 2007
- By Carrot or by Stick: Cognitive Reinforcement Learning in ParkinsonismScience, 2004
- Discrete Coding of Reward Probability and Uncertainty by Dopamine NeuronsScience, 2003
- Optimal Short-Term Population Coding: When Fisher Information FailsNeural Computation, 2002
- A Neural Substrate of Prediction and RewardScience, 1997
- A Theory of Dopamine Function and Its Role in Cognitive Deficits in SchizophreniaSchizophrenia Bulletin, 1993
- Some informational aspects of visual perception.Psychological Review, 1954