Intrinsically Motivated Hierarchical Skill Learning in Structured Environments
Authors:
- 17 May 2010
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Autonomous Mental Development
- Vol. 2 (2) , 132-143
Abstract
We present a framework for intrinsically motivated developmental learning of abstract skill hierarchies by reinforcement learning agents in structured environments. Long-term learning of skill hierarchies can drastically improve an agent's efficiency in solving ensembles of related tasks in a complex domain. In structured domains composed of many features, understanding the causal relationships between actions and their effects on different features of the environment can greatly facilitate skill learning. Using Bayesian network structure (learning techniques and structured dynamic programming algorithms), we show that reinforcement learning agents can learn incrementally and autonomously both the causal structure of their environment and a hierarchy of skills that exploit this structure. Furthermore, we present a novel active learning scheme that employs intrinsic motivation to maximize the efficiency with which this structure is learned. As new structure is acquired using an agent's current set of skills, more complex skills are learned, which in turn allow the agent to discover more structure, and so on. This bootstrapping property makes our approach a developmental learning process that results in steadily increasing domain knowledge and behavioral complexity as an agent continues to explore its environment.Keywords
This publication has 17 references indexed in Scilit:
- Knows what it knows: a framework for self-aware learningMachine Learning, 2010
- Active Learning of Dynamic Bayesian Networks in Markov Decision Processes2007
- Learning the structure of Factored Markov Decision Processes in reinforcement learning problems2006
- Maximizing Learning Progress: An Internal Reward System for Development2004
- Stochastic dynamic programming with factored representationsArtificial Intelligence, 2000
- 10.1162/153244303765208377Applied Physics Letters, 2000
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999
- Improving generalization with active learningMachine Learning, 1994
- A model for reasoning about persistence and causationComputational Intelligence, 1989
- A theory of the learnableCommunications of the ACM, 1984