Toward Model-Assisted Safe Reinforcement Learning for Data Center Cooling Control: A Lyapunov-based Approach
Open Access
- 16 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 333-346
Abstract
This paper considers intelligent data center cooling control via the Deep Reinforcement Learning (DRL) approach to improve data center sustainability. Existing DRL-based controllers are trained with a simplified data hall thermodynamic model which assumes uniform room temperature distribution. This assumption is not valid for a real-world data center with highly nonuniform temperature distribution. Furthermore, most of them cannot guarantee thermal safety during the DRL learning process. To bridge these gaps, we propose LyaSafe, a model-assisted safe DRL approach for data center cooling control. To address the safety evaluation issue, we develop a coupled model that combines a differentiable surrogate data hall thermodynamics model with the energy model. It can simulate both data hall temperature distribution and the facility energy consumption. To address safe learning, we introduce a novel constrained Markov Decision Process (CMDP) formulation for data center cooling control by considering the Rack Cooling Index (RCI), the best-practice metric for evaluating compliance with ASHRAE data center thermal guidelines. The objective is to minimize data center carbon footprints while regulating the RCI within a threshold. We first derive the safety set based on the concept of the virtual queue and Lyapunov stability theory. Next, we rectify unsafe actions from the DRL agent by projecting them to the safety set. We evaluate LyaSafe in a data center hosting 20 racks and 299 servers. Evaluation results show that LyaSafe can ensure strict safety during the DRL learning while achieving up to 50 metric tons of annual carbon emission savings using Singapore’s statistics. Moreover, we conduct root cause analysis for the savings, revealing the importance of joint control of the data hall and the chiller plant.Keywords
Funding Information
- National Natural Science Foundation, China (No. 62262026, No. GJJ211111)
- Ministry of Education, Singapore (Tier 1 of RT14/22 and RG96/20)
- Nanyang Technological University, Singapore (No. NTU?ACE2020-01)
- National Research Foundation, Singapore (NRF2020NRF-CG001-027)
This publication has 15 references indexed in Scilit:
- Building HVAC Scheduling Using Reinforcement Learning via Neural Network Based Model Approximation2019
- Transforming Cooling Optimization for Green Data Center via Deep Reinforcement LearningIEEE Transactions on Cybernetics, 2019
- DeepEE: Joint Optimization of Job Scheduling and Cooling Control for Data Center Energy Efficiency Using Deep Reinforcement Learning2019
- Who limits the resource efficiency of my datacenter2019
- Building energy simulation coupled with CFD for indoor environment: A critical review and recent applicationsEnergy and Buildings, 2018
- United States Data Center Energy Usage Report2016
- Data center cooling management and analysis - a model based approach2012
- Modeling and Control for Cooling Management of Data Centers With Hot Aisle Containment2011
- Supervisory and Optimal Control of Building HVAC Systems: A ReviewHVAC&R Research, 2008
- PID autotuner and its application in HVAC systems2001