Optimizing agent behavior over long time scales by transporting value
Open Access
- 19 November 2019
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 10 (1) , 1-12
Abstract
Humans prolifically engage in mental time travel. We dwell on past actions and experience satisfaction or regret. More than storytelling, these recollections change how we act in the future and endow us with a computationally important ability to link actions and consequences across spans of time, which helps address the problem of long-term credit assignment: the question of how to evaluate the utility of actions within a long-duration behavioral sequence. Existing approaches to credit assignment in AI cannot solve tasks with long delays between actions and consequences. Here, we introduce a paradigm where agents use recall of specific memories to credit past actions, allowing them to solve problems that are intractable for existing algorithms. This paradigm broadens the scope of problems that can be investigated in AI and offers a mechanistic account of behaviors that may inspire models in neuroscience, psychology, and behavioral economics.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Human-level control through deep reinforcement learningNature, 2015
- Episodic Future Thinking Reduces Reward Delay Discounting through an Enhancement of Prefrontal-Mediotemporal InteractionsNeuron, 2010
- Using Imagination to Understand the Neural Basis of Episodic MemoryThe Journal of Neuroscience, 2007
- Remembering the past to imagine the future: the prospective brainNature Reviews Neuroscience, 2007
- Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomesPsychological Research, 2007
- A Dual-Self Model of Impulse ControlAmerican Economic Review, 2006
- Delaying execution of intentions: overcoming the costs of interruptionsApplied Cognitive Psychology, 2004
- Time Discounting and Time Preference: A Critical ReviewJournal of Economic Literature, 2002
- Neural Networks and the Bias/Variance DilemmaNeural Computation, 1992
- Cognitive maps in rats and men.Psychological Review, 1948