Behavior Priors for Efficient Reinforcement Learning
Authors:
- Dhruva TirumalaDTDhruva Tirumala
- Alexandre GalashovAGAlexandre Galashov
- Hyeonwoo NohHNHyeonwoo Noh
- Leonard HasencleverLHLeonard Hasenclever
- Razvan PascanuRPRazvan Pascanu
- Jonathan SchwarzJSJonathan Schwarz
- Guillaume DesjardinsGDGuillaume Desjardins
- Wojciech Marian CzarneckiWCWojciech Marian Czarnecki
- Arun AhujaAAArun Ahuja
- Yee Whye TehYTYee Whye Teh
- Nicolas HeessNHNicolas Heess
Preprint
Abstract
As we deploy reinforcement learning agents to solve increasingly challenging problems, methods that allow us to inject prior knowledge about the structure of the world and effective solution strategies becomes increasingly important. In this work we consider how information and architectural constraints can be combined with ideas from the probabilistic modeling literature to learn behavior priors that capture the common movement and interaction patterns that are shared across a set of related tasks or contexts. For example the day-to day behavior of humans comprises distinctive locomotion and manipulation patterns that recur across many different situations and goals. We discuss how such behavior patterns can be captured using probabilistic trajectory models and how these can be integrated effectively into reinforcement learning schemes, e.g.\ to facilitate multi-task and transfer learning. We then extend these ideas to latent variable models and consider a formulation to learn hierarchical priors that capture different aspects of the behavior in reusable modules. We discuss how such latent variable formulations connect to related work on hierarchical reinforcement learning (HRL) and mutual information and curiosity based objectives, thereby offering an alternative perspective on existing ideas. We demonstrate the effectiveness of our framework by applying it to a range of simulated continuous control domains.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Grandmaster level in StarCraft II using multi-agent reinforcement learningNature, 2019
- Mental labourNature Human Behaviour, 2018
- Probabilistic inference for determining options in reinforcement learningMachine Learning, 2016
- Empowerment–An Introduction2014
- Thermodynamics as a theory of decision-making with information-processing costsProceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2013
- An information-theoretic approach to curiosity-driven reinforcement learningTheory in Biosciences, 2012
- Information Theory of Decisions and Actions2010
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999
- Rational choice and the structure of the environment.Psychological Review, 1956