Linear reinforcement learning in planning, grid fields, and cognitive control
Open Access
- 16 August 2021
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 12 (1) , 1-20
Abstract
It is thought that the brain’s judicious reuse of previous computation underlies our ability to plan flexibly, but also that inappropriate reuse gives rise to inflexibilities like habits and compulsion. Yet we lack a complete, realistic account of either. Building on control engineering, here we introduce a model for decision making in the brain that reuses a temporally abstracted map of future events to enable biologically-realistic, flexible choice at the expense of specific, quantifiable biases. It replaces the classic nonlinear, model-based optimization with a linear approximation that softly maximizes around (and is weakly biased toward) a default policy. This solution demonstrates connections between seemingly disparate phenomena across behavioral neuroscience, notably flexible replanning with biases and cognitive control. It also provides insight into how the brain can represent maps of long-distance contingencies stably and componentially, as in entorhinal response fields, and exploit them to guide choice even under changing goals.Keywords
All Related Versions
This publication has 71 references indexed in Scilit:
- Habits, action sequences and reinforcement learningEuropean Journal of Neuroscience, 2012
- Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement LearningPLOS Computational Biology, 2011
- Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed ProcessesPLOS Computational Biology, 2011
- Model-Based Influences on Humans' Choices and Striatal Prediction ErrorsNeuron, 2011
- The influence of reward associations on conflict processing in the Stroop taskCognition, 2010
- Decision making and the avoidance of cognitive demand.Journal of Experimental Psychology: General, 2010
- Hierarchically organized behavior and its neural foundations: A reinforcement learning perspectiveCognition, 2009
- Efficient computation of optimal actionsProceedings of the National Academy of Sciences, 2009
- Microstructure of a spatial map in the entorhinal cortexNature, 2005
- Simplified neuron model as a principal component analyzerJournal of Mathematical Biology, 1982