Adaptive Policy Regularization for Offline-to-Online Reinforcement Learning in HVAC Control
Open Access
- 29 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Abstract
Reinforcement learning (RL)-based control methods have been extensively studied to improve building heating, ventilation, and air conditioning (HVAC) efficiency. Data-driven approaches demonstrate better transferability and scalability, making them useful in real-world applications. Most prior works focus on online learning requiring simulators or models of environment dynamics. However, transferring thermal simulators between environments is inefficient in practice. We build on recent works that employ offline training on static datasets from unknown policies. Pure offline RL is constrained by the replay buffer's distribution, we propose using offline-to-online RL to enhance pre-trained offline models through online adaptation to distribution shifts. We show that direct online fine-tuning deteriorates performance on offline policies. To address this, we propose automatically tuning the actor's regularization during training to optimize the exploration-exploitation tradeoff. Specifically, we leverage simple moving averages of mean Q-values sampled throughout training. Simulation experiments demonstrate our method outperforms state-of-the-art approaches under various conditions, improving performance by 32.9% and enhancing pre-trained models' capabilities online.Keywords
Funding Information
- ACE, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA (#2023-JU-3134)
This publication has 11 references indexed in Scilit:
- B2RL2022
- Safe HVAC Control via Batch Reinforcement Learning2022
- Safe Building HVAC Control via Batch Reinforcement LearningIEEE Transactions on Sustainable Computing, 2022
- Sinergym2021
- Reinforcement learning for building controls: The opportunities and challengesApplied Energy, 2020
- CityLearn v1.02019
- A review of reinforcement learning methodologies for controlling occupant comfort in buildingsSustainable Cities and Society, 2019
- Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learningEnergy and Buildings, 2019
- Reinforcement learning for optimal control of low exergy buildingsApplied Energy, 2015
- Boosted Bellman Residual Minimization Handling Expert Demonstrations2014