Induction of Subgoal Automata for Reinforcement Learning
Preprint
Abstract
In this work we present ISA, a novel approach for learning and exploiting subgoals in reinforcement learning (RL). Our method relies on inducing an automaton whose transitions are subgoals expressed as propositional formulas over a set of observable events. A state-of-the-art inductive logic programming system is used to learn the automaton from observation traces perceived by the RL agent. The reinforcement learning and automaton learning processes are interleaved: a new refined automaton is learned whenever the RL agent generates a trace not recognized by the current automaton. We evaluate ISA in several gridworld problems and show that it performs similarly to a method for which automata are given in advance. We also show that the learned automata can be exploited to speed up convergence through reward shaping and transfer learning across multiple tasks. Finally, we analyze the running time and the number of traces that ISA needs to learn an automata, and the impact that the number of observable events has on the learner's performance.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Joint Inference of Reward Machines and Policies for Reinforcement LearningProceedings of the International Conference on Automated Planning and Scheduling, 2020
- LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning2019
- A general reinforcement learning algorithm that masters chess, shogi, and Go through self-playScience, 2018
- Computing Hierarchical Finite State Controllers With Classical PlanningJournal of Artificial Intelligence Research, 2018
- Iterative Learning of Answer Set Programs from Context Dependent ExamplesTheory and Practice of Logic Programming, 2016
- Automatic Derivation of Memoryless Policies and Finite-State Controllers Using Classical PlannersProceedings of the International Conference on Automated Planning and Scheduling, 2009
- Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learningArtificial Intelligence, 1999