Hierarchical Reinforcement Learning with Targeted Causal Interventions
Preprint
Abstract
Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erdős-Rényi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Reward Machines: Exploiting Reward Function Structure in Reinforcement LearningJournal of Artificial Intelligence Research, 2022
- Causality-Driven Hierarchical Structure Discovery for Reinforcement Learning2022
- Neuro-Symbolic Reinforcement Learning with First-Order Logic2021
- MineRL: A Large-Scale Dataset of Minecraft Demonstrations2019
- Hierarchical Reinforcement Learning with the MAXQ Value Function DecompositionJournal of Artificial Intelligence Research, 2000
- A Formal Basis for the Heuristic Determination of Minimum Cost PathsIEEE Transactions on Systems Science and Cybernetics, 1968