Computational evidence for hierarchically structured reinforcement learning in humans
- 23 November 2020
- journal article
- research article
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 117 (47) , 29381-29389
Abstract
Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine-learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models—a classic RL, a hierarchical RL, and a hierarchical Bayesian model—and compared their behavior to human results. While the flat RL model captured some aspects of participants’ sensitivity to outcome values, and the hierarchical Bayesian model captured some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments and opens the avenue for future research in this field.Keywords
All Related Versions
Funding Information
- NIH (1R01MH119383-01)
This publication has 57 references indexed in Scilit:
- Approximate Bayesian ComputationPLOS Computational Biology, 2013
- Neural Basis of Reinforcement Learning and Decision MakingAnnual Review of Neuroscience, 2012
- Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-MakingPLOS Biology, 2012
- A Neural Signature of Hierarchical Reinforcement LearningNeuron, 2011
- Mechanisms of Hierarchical Reinforcement Learning in Cortico-Striatal Circuits 2: Evidence from fMRICerebral Cortex, 2011
- Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational AnalysisCerebral Cortex, 2011
- Hierarchically organized behavior and its neural foundations: A reinforcement learning perspectiveCognition, 2009
- Is the rostro-caudal axis of the frontal lobe hierarchical?Nature Reviews Neuroscience, 2009
- Hierarchical Bayesian inference in the visual cortexJournal of the Optical Society of America A, 2003
- A Neural Substrate of Prediction and RewardScience, 1997