Curious Hierarchical Actor-Critic Reinforcement Learning
Authors:
Preprint
Abstract
Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity. However, there is a lack of approaches that combine these paradigms, and it is currently unknown whether curiosity also helps to perform the hierarchical abstraction. As a novelty and scientific contribution, we tackle this issue and develop a method that combines hierarchical reinforcement learning with curiosity. Herein, we extend a contemporary hierarchical actor-critic approach with a forward model to develop a hierarchical notion of curiosity. We demonstrate in several continuous-space environments that curiosity can more than double the learning performance and success rates for most of the investigated benchmarking problems. We also provide our source code and a supplementary video.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- From Semantics to Execution: Integrating Action Planning With Reinforcement Learning for Robotic Causal Problem-SolvingFrontiers in Robotics and AI, 2019
- Towards a neuroscience of active sampling and curiosityNature Reviews Neuroscience, 2018
- Hierarchical Active Inference: A Theory of Motivated ControlTrends in Cognitive Sciences, 2018
- Intrinsically motivated model learning for developing curious robotsArtificial Intelligence, 2017
- Exploration Behaviors, Body Representations, and Simulation Processes for the Development of Cognition in Artificial AgentsFrontiers in Robotics and AI, 2016
- Toward a Unified Sub-symbolic Computational Theory of CognitionFrontiers in Psychology, 2016
- Model-based hierarchical reinforcement learning and human action controlPhilosophical Transactions of the Royal Society B: Biological Sciences, 2014
- Action understanding and active inferenceBiological Cybernetics, 2011
- Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)IEEE Transactions on Autonomous Mental Development, 2010