Feedback for Improved Hierarchical Reinforcement Learning With Timed Subgoals
- 4 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Control Systems Letters
- Vol. 9, 2501-2506
Abstract
Hierarchical reinforcement learning (RL) aims to improve sample efficiency by decomposing complex long-horizon tasks into fast low-level myopic and slower high-level non-myopic subtasks. However, the unilateral nested policy structure in current goal-conditioned hierarchical RL (HRL) methods sets subgoals at the high level without considering feedback from the low level, which significantly degrades the performance of high-level subgoal generation and sampling efficiency. Hindsight action relabeling further weakens subgoal settings by submitting to low-level reachability. Inspired by feedback control of dynamic systems, we present Feedback for Improved HRL with Timed Subgoals (FIHTS), a mechanism allowing feedback control of subgoal generation for improved HRL. Unlike current HRL, FIHTS enables both the high level to set subgoals and the low level to receive rewards based on subgoal achievement. Our experiments in various challenging dynamic RL environments show that our FIHTS method achieves higher success rates with higher sample efficiency than existing subgoal-based HRL methods.Keywords
Funding Information
- Department of the Air Force (FA8750-23-C-0509)
This publication has 8 references indexed in Scilit:
- Explicit-Implicit Subgoal Planning for Long-Horizon Tasks With Sparse RewardsIEEE Transactions on Automation Science and Engineering, 2025
- Goal-Guided Reinforcement Learning: Leveraging Large Language Models for Long-Horizon Task Decomposition2025
- Predictive Reachability for Embodiment Selection in Mobile Manipulation BehaviorsIEEE Robotics and Automation Letters, 2025
- MENTOR: Guiding Hierarchical Reinforcement Learning With Human Feedback and Dynamic Distance ConstraintIEEE Transactions on Emerging Topics in Computational Intelligence, 2025
- Guided Cooperation in Hierarchical Reinforcement Learning via Model-Based RolloutIEEE Transactions on Neural Networks and Learning Systems, 2024
- State-Conditioned Adversarial Subgoal GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks2022
- Model-free Control Design Using Policy Gradient Reinforcement Learning in LPV Framework2021