Temporal Abstraction in Reinforcement Learning with the Successor Representation
Preprint
Abstract
Reasoning at multiple levels of temporal abstraction is one of the key attributes of intelligence. In reinforcement learning, this is often modeled through temporally extended courses of actions called options. Options allow agents to make predictions and to operate at different levels of abstraction within an environment. Nevertheless, approaches based on the options framework often start with the assumption that a reasonable set of options is known beforehand. When this is not the case, there are no definitive answers for which options one should consider. In this paper, we argue that the successor representation (SR), which encodes states based on the pattern of state visitation that follows them, can be seen as a natural substrate for the discovery and use of temporal abstractions. To support our claim, we take a big picture view of recent results, showing how the SR can be used to discover options that facilitate either temporally-extended exploration or planning. We cast these results as instantiations of a general framework for option discovery in which the agent's representation is used to identify useful options, which are then used to further improve its representation. This results in a virtuous, never-ending, cycle in which both the representation and the options are constantly refined based on each other. Beyond option discovery itself, we also discuss how the SR allows us to augment a set of options into a combinatorially large counterpart without additional learning. This is achieved through the combination of previously learned options. Our empirical evaluation focuses on options discovered for exploration and on the use of the SR to combine them. The results of our experiments shed light on important design decisions involved in the definition of options and demonstrate the synergy of different methods based on the SR, such as eigenoptions and the option keyboard.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Reward-respecting subtasks for model-based reinforcement learningArtificial Intelligence, 2023
- Fast reinforcement learning with generalized policy updatesProceedings of the National Academy of Sciences, 2020
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- The successor representation in human reinforcement learningNature Human Behaviour, 2017
- Active learning of inverse models with intrinsically motivated goal exploration in robotsRobotics and Autonomous Systems, 2013
- Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion mapsProceedings of the National Academy of Sciences, 2005
- Slow Feature Analysis: Unsupervised Learning of InvariancesNeural Computation, 2002
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999
- Improving Generalization for Temporal Difference Learning: The Successor RepresentationNeural Computation, 1993
- Bounds on the cover timeJournal of Theoretical Probability, 1989