Abstract
Hierarchical reinforcement learning (RL) aims to improve sample efficiency by decomposing complex long-horizon tasks into fast low-level myopic and slower high-level non-myopic subtasks. However, the unilateral nested policy structure in current goal-conditioned hierarchical RL (HRL) methods sets subgoals at the high level without considering feedback from the low level, which significantly degrades the performance of high-level subgoal generation and sampling efficiency. Hindsight action relabeling further weakens subgoal settings by submitting to low-level reachability. Inspired by feedback control of dynamic systems, we present Feedback for Improved HRL with Timed Subgoals (FIHTS), a mechanism allowing feedback control of subgoal generation for improved HRL. Unlike current HRL, FIHTS enables both the high level to set subgoals and the low level to receive rewards based on subgoal achievement. Our experiments in various challenging dynamic RL environments show that our FIHTS method achieves higher success rates with higher sample efficiency than existing subgoal-based HRL methods.
Funding Information
  • Department of the Air Force (FA8750-23-C-0509)

This publication has 8 references indexed in Scilit: