Program-Based Strategy Induction for Reinforcement Learning
Preprint
Abstract
Typical models of learning assume incremental estimation of continuously-varying decision variables like expected rewards. However, this class of models fails to capture more idiosyncratic, discrete heuristics and strategies that people and animals appear to exhibit. Despite recent advances in strategy discovery using tools like recurrent networks that generalize the classic models, the resulting strategies are often onerous to interpret, making connections to cognition difficult to establish. We use Bayesian program induction to discover strategies implemented by programs, letting the simplicity of strategies trade off against their effectiveness. Focusing on bandit tasks, we find strategies that are difficult or unexpected with classical incremental learning, like asymmetric learning from rewarded and unrewarded trials, adaptive horizon-dependent random exploration, and discrete state switching.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Choice-confirmation bias and gradual perseveration in human reinforcement learning.Behavioral Neuroscience, 2023
- Linear reinforcement learning in planning, grid fields, and cognitive controlNature Communications, 2021
- Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resourcesBehavioral and Brain Sciences, 2019
- An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervalsNature Communications, 2018
- Reminders of past choices bias decisions for reward in humansNature Communications, 2017
- Memory states influence value-based decisions.Journal of Experimental Psychology: General, 2016
- Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal targetNature Neuroscience, 2016
- Reliance on small samples, the wavy recency effect, and similarity-based learning.Psychological Review, 2015
- Humans use directed and random exploration to solve the explore–exploit dilemma.Journal of Experimental Psychology: General, 2014
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998