A hierarchical multi‑timescale reinforcement learning framework for hybrid energy storage management with aging‑aware constraint adaptation
Open Access
- 4 June 2026
- journal article
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
Abstract
Hybrid energy storage systems (HESS), combining batteries and supercapacitors, offer an effective solution for simultaneously addressing energy and power demands in modern energy systems. However, the performance and lifetime of these systems are strongly dependent on the employed energy management strategy, which must balance fast power regulation, safety enforcement, and long-term battery health. Existing reinforcement learning-based approaches often adopt flat control architectures that entangle short-term decision-making with long-term aging effects, leading to training instability, limited interpretability, and non‑stationary reward structures. This paper proposes a hierarchical multi‑timescale reinforcement learning framework for energy management of battery-supercapacitor hybrid systems with explicit aging‑aware constraint adaptation. The proposed architecture decomposes the control problem into three coordinated layers: a fast safety‑critical layer that enforces hard operational constraints, an intermediate reinforcement learning layer based on Proximal Policy Optimization (PPO) for real‑time power allocation, and a slow supervisory layer that estimates battery state‑of‑health (SOH) using a Gaussian Process Regression (GPR) aging model. Instead of directly embedding aging costs into the learning objective, battery degradation is incorporated through SOH‑dependent adaptive current limits, ensuring quasi‑stationary learning dynamics and safe long‑term operation. Comprehensive simulation studies are conducted under nominal, progressive aging, and safety‑critical operating conditions. Under nominal SOH, the proposed framework maintains battery and supercapacitor SOC deviations within ± 3% of their reference values while keeping battery current below the nominal 125 A limit. During aging progression, the method achieves a lower time‑averaged squared battery current and reduced cumulative energy throughput compared to reward‑based SAC and flat RL baselines, resulting in a measurably slower SOH decline for equivalent stress levels. Moreover, under aggressive pulsed loads at reduced SOH, the proposed approach yields the lowest constraint activation frequency, demonstrating superior intrinsic safety and robustness. Overall, the results confirm that hierarchical time‑scale separation enables stable, aging‑aware, and physically consistent energy management without compromising learning performance.Keywords
This publication has 41 references indexed in Scilit:
- Deep reinforcement learning for joint dispatch of battery storage and gas turbines in renewable-powered microgridsEnergy and AI, 2025
- Optimizing energy management of smart grid using reinforcement learning aided by surrogate models built using physics-informed neural networksApplied Energy, 2025
- Battery aging-aware adaptive model predictive control based on coupled semi-empirical electro-thermal and aging modelsApplied Energy, 2025
- Deep Q-network based battery energy storage system control strategy with charging/discharging times consideredApplied Energy, 2025
- Reinforcement learning-based energy management for hybrid electric vehicles: A comprehensive up-to-date review on methods, challenges, and research gapsEnergy and AI, 2025
- Deep reinforcement learning for optimal microgrid energy management with renewable energy and electric vehicle integrationApplied Soft Computing, 2025
- Reinforcement learning for adaptive battery management of structural health monitoring IoT sensor networkApplied Energy, 2025
- Energy management for scalable battery swapping stations: A deep reinforcement learning and mathematical optimization cascade approachApplied Energy, 2024
- Hierarchical energy management system for multi-microgrid coordination with demand-side managementApplied Energy, 2023
- Hierarchical reinforcement learning based energy management strategy of plug-in hybrid electric vehicle for ecological car-following processApplied Energy, 2023