Learning Compositional Neural Programs for Continuous Control
Preprint
Abstract
We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple levels of abstraction. The key insight is that the self-models enable planning by imagination, obviating the need for interaction with the world when learning higher-level compositional programs. To accomplish the third stage of learning, we extend the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural programmer-interpreters. We empirically show that AlphaNPI-X can effectively learn to tackle challenging sparse manipulation tasks, such as stacking multiple blocks, where powerful model-free baselines fail.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Mastering Atari, Go, chess and shogi by planning with a learned modelNature, 2020
- Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning2020
- Learning dexterous in-hand manipulationThe International Journal of Robotics Research, 2019
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots2018
- Neural Task Programming: Learning to Generalize Across Hierarchical Tasks2018
- The Option-Critic ArchitectureProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Efficient Solution Algorithms for Factored MDPsJournal of Artificial Intelligence Research, 2003
- Model predictive control: Theory and practice—A surveyAutomatica, 1989