XDXianzhong Ding

Xianzhong Ding

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Xianzhong Ding has a total of 22 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Scalable and Efficient Reinforcement Learning for Virtual Machine Rescheduling in Cloud Data Centers

    by Xianzhong Ding, Yunkai Zhang, Binbin Chen, Donghao Ying, Tieying Zhang, Jianjun Chen, Lei Zhang, Alberto Cerpa, Wan Du

    Managing a vast number of virtual machines (VMs) efficiently is a critical challenge in modern large-scale data centers. The continuous creation and termination of VMs lead to resource fragmentation across physical machines (PMs), necessitating periodic VM rescheduling to optimize resource utilization. Despite its significance, VM rescheduling has received limited attention in the literature. A key challenge is that, unlike conventional combinatorial optimization problems, the efficiency of rescheduling algorithms is heavily impacted by inference time, as VM states evolve dynamically during execution. This scalability bottleneck hampers existing methods. To address this, we propose VMR$^{2}$L, a reinforcement learning framework tailored for VM rescheduling. VMR$^{2}$L integrates a two-stage decision-making process to accommodate complex operational constraints, a feature extraction mechanism that captures critical relational information for rescheduling, and a risk-aware evaluation strategy that enables users to balance execution speed and rescheduling accuracy. Extensive experiments using real-world data from a production-scale data center demonstrate that VMR$^{2}$L achieves near-optimal performance while reducing inference time to a matter of seconds.


  • Open Access

    Towards VM Rescheduling Optimization Through Deep Reinforcement Learning

    by Xianzhong Ding, Yunkai Zhang, Binbin Chen, Donghao Ying, Tieying Zhang, Jianjun Chen, Lei Zhang, Alberto Cerpa, Wan Du


  • A Safe and Data-Efficient Model-Based Reinforcement Learning System for HVAC Control

    by Xianzhong Ding, Zhiyu An, Arya Rathee, Wan Du

    Model-Based Reinforcement Learning (MBRL) is widely studied for Heating, Ventilation, and Air Conditioning (HVAC) control in buildings. One of the critical challenges is the large amount of data required to effectively train neural networks for modeling building dynamics. This paper presents CLUE, an MBRL system for HVAC control in buildings. CLUE optimizes HVAC operations by integrating a Gaussian Process (GP) model to model building dynamics with uncertainty awareness. CLUE utilizes GP to predict state transitions as Gaussian distributions, effectively capturing prediction uncertainty and enhancing decision-making under sparse data conditions. Our approach employs a meta-kernel learning technique to efficiently set GP kernel hyperparameters using domain knowledge from diverse buildings. This drastically reduces the data requirements typically associated with GP models in HVAC applications. Additionally, CLUE incorporates these uncertainty estimates into a Model Predictive Path Integral (MPPI) algorithm, enabling the selection of safe, energy-efficient control actions. This uncertainty-aware control strategy evaluates and selects action trajectories based on their predicted impact on energy consumption and human comfort, optimizing operations even under uncertain conditions. Extensive simulations in a five-zone office building demonstrate that CLUE reduces the required training data from hundreds of days to just seven while maintaining robust control performance. It reduces comfort violations by an average of 12.07% compared to existing MBRL methods, without compromising on energy efficiency. Our code and dataset are available at https://github.com/ryeii/CLUE.


  • Open Access

    Optimizing Irrigation Efficiency using Deep Reinforcement Learning in the Field

    by Xianzhong Ding, Wan Du

    Agricultural irrigation is a significant contributor to freshwater consumption. However, the current irrigation systems used in the field are not efficient. They rely mainly on soil moisture sensors and the experience of growers but do not account for future soil moisture loss. Predicting soil moisture loss is challenging because it is influenced by numerous factors, including soil texture, weather conditions, and plant characteristics. This article proposes a solution to improve irrigation efficiency, which is called DRLIC (deep reinforcement learning for irrigation control). DRLIC is a sophisticated irrigation system that uses deep reinforcement learning (DRL) to optimize its performance. The system employs a neural network, known as the DRL control agent, which learns an optimal control policy that considers both the current soil moisture measurement and the future soil moisture loss. We introduce an irrigation reward function that enables our control agent to learn from previous experiences. However, there may be instances in which the output of our DRL control agent is unsafe, such as irrigating too much or too little. To avoid damaging the health of the plants, we implement a safety mechanism that employs a soil moisture predictor to estimate the performance of each action. If the predicted outcome is deemed unsafe, we perform a relatively conservative action instead. To demonstrate the real-world application of our approach, we develop an irrigation system that comprises sprinklers, sensing and control nodes, and a wireless network. We evaluate the performance of DRLIC by deploying it in a testbed consisting of six almond trees. During a 15-day in-field experiment, we compare the water consumption of DRLIC with a widely used irrigation scheme. Our results indicate that DRLIC outperforms the traditional irrigation method by achieving water savings of up to 9.52%.


  • Open Access

    Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control

    by Zhiyu An, Xianzhong Ding, Wan Du

    Recent research has shown the potential of Model-based Reinforcement Learning (MBRL) to enhance energy efficiency of Heating, Ventilation, and Air Conditioning (HVAC) systems. However, existing methods rely on black-box thermal dynamics models and stochastic optimizers, lacking reliability guarantees and posing risks to occupant health. In this work, we overcome the reliability bottleneck by redesigning HVAC controllers using decision trees extracted from existing thermal dynamics models and historical data. Our decision tree-based policies are deterministic, verifiable, interpretable, and more energy-efficient than current MBRL methods. First, we introduce a novel verification criterion for RL agents in HVAC control based on domain knowledge. Second, we develop a policy extraction procedure that produces a verifiable decision tree policy. We found that the high dimensionality of the thermal dynamics model input hinders the efficiency of policy extraction. To tackle the dimensionality challenge, we leverage importance sampling conditioned on historical data distributions, significantly improving policy extraction efficiency. Lastly, we present an offline verification algorithm that guarantees the reliability of a control policy. Extensive experiments show that our method saves 68.4% more energy and increases human comfort gain by 14.8% compared to the state-of-the-art method, in addition to an 1127× reduction in computation overhead. Our code and data are available at https://github.com/ryeii/Veri_HVAC.


  • Multi-Zone HVAC Control With Model-Based Deep Reinforcement Learning

    by Xianzhong Ding, Alberto Cerpa, Wan Du

    The application of reinforcement learning in controlling Heating, Ventilation, and Air Conditioning (HVAC) systems has been extensively researched. Existing studies primarily focus on Model-Free Reinforcement Learning (MFRL), which involves trial-and-error interactions with real buildings to train the agent. However, MFRL encounters a significant challenge: it requires a large amount of training data to achieve satisfactory performance. While simulation models have been used to generate training data and expedite the training process, they necessitate high-fidelity building models that are difficult to calibrate. As a result, Model-Based Reinforcement Learning (MBRL) has been employed for HVAC control. Although MBRL demonstrates remarkable sample efficiency, it often falls short in terms of asymptotic control performance, particularly in achieving substantial energy savings while ensuring occupants’ thermal comfort. In this study, we conduct experiments to analyze the limitations of current MBRL-based HVAC control methods, focusing on model uncertainty and controller effectiveness. Leveraging the insights gained from these experiments, we develop, an innovative MBRL-based HVAC control system that combines high control performance with exceptional sample efficiency. learns the dynamics of the building by employing an ensemble of environment-conditioned neural networks and utilizes a novel control method called Model Predictive Path Integral (MPPI) for HVAC control. MPPI generates candidate action sequences using an importance sampling weighted algorithm, which is well-suited for multi-zone buildings with high state and action dimensions. We evaluate using EnergyPlus simulations in a five-zone office building, and the results demonstrate that achieves 8.23% higher energy savings compared to the state-of-the-art MBRL solution while maintaining comparable thermal comfort. Moreover, significantly reduces the required training data set by an order of magnitude (10.52 $\times$ ) while delivering performance on par with MFRL approaches. Note to Practitioners —Our research addresses a critical challenge in HVAC control, offering an innovative solution to enhance the data efficiency of HVAC systems while optimizing energy usage. Traditional approaches, such as Model-Free Reinforcement Learning, often require a large volume of real-world data. Our primary focus is improving the effectiveness of HVAC control, a vital aspect of building management that directly affects energy consumption and occupant well-being. We introduce, a Model-Based Reinforcement Learning system designed to significantly improve energy savings while maintaining thermal comfort. achieves remarkable results, offering exceptional sample efficiency and substantially reducing the required training data. Our research leverages an ensemble of environment-conditioned neural networks and employs Model Predictive Path Integral in HVAC control. While presents notable benefits, it also has limitations. Further research and development are required to optimize its performance across different building environments and specific use cases. Future directions should focus on addressing the safety challenges associated with real-world deployment. Beyond HVAC control, the principles and methods explored in this research have potential applications in various automation domains, such as robotics, industrial automation, and manufacturing processes.


  • Open Access

    Exploring Deep Reinforcement Learning for Holistic Smart Building Control

    by Xianzhong Ding, Alberto Cerpa, Wan Du

    In recent years, the focus has been on enhancing user comfort in commercial buildings while cutting energy costs. Efforts have mainly centered on improving HVAC systems, the central control system. However, it’s evident that HVAC alone can’t ensure occupant comfort. Lighting, blinds, and windows, often overlooked, also impact energy use and comfort. This paper introduces a holistic approach to managing the delicate balance between energy efficiency and occupant comfort in commercial buildings. We present OCTOPUS , a system employing a deep reinforcement learning (DRL) framework using data-driven techniques to optimize control sequences for all building subsystems, including HVAC, lighting, blinds, and windows. OCTOPUS ’s DRL architecture features a unique reward function facilitating the exploration of tradeoffs between energy usage and user comfort, effectively addressing the high-dimensional control problem resulting from interactions among these four building subsystems. To meet data training requirements, we emphasize the importance of calibrated simulations that closely replicate target-building operational conditions. We train OCTOPUS using 10-year weather data and a calibrated building model in the EnergyPlus simulator. Extensive simulations demonstrate that OCTOPUS achieves substantial energy savings, outperforming state-of-the-art rule-based and DRL-based methods by 14.26% and 8.1%, respectively, in a LEED Gold Certified building while maintaining desired human comfort levels.


  • Open Access

    CLUE: Safe Model-Based RL HVAC Control Using Epistemic Uncertainty Estimation

    by Zhiyu An, Xianzhong Ding, Arya Rathee, Wan Du

    Model-Based Reinforcement Learning (MBRL) has been widely studied for Heating, Ventilation, and Air Conditioning (HVAC) control in buildings. One of the fundamental problems is the large amount of data required to train a neural network for building dynamics modeling. In this paper, we developed CLUE, a safe MBRL HVAC control approach that can achieve low human comfort violation with a dynamics model trained on a small dataset. We used Gaussian Process (GP) as the building dynamics model, which provides the uncertainty of each output. The uncertainty result is then integrated into a safe HVAC control algorithm. Although GP has been studied for HVAC control, this work provides a data-efficient GP modeling method. We designed a novel meta kernel learning technique that incorporates domain knowledge from historical data of multiple buildings to set the GP kernel hyperparameters. Our method can significantly reduce the amount of data required for GP hyperparameter setting. Furthermore, we incorporate the GP-based uncertainty into a Model Predictive Path Integral (MPPI) process to find a safe, energy-efficient action for each control cycle. We generate a large number of action trajectories by the GP building dynamics model, and find the optimal trajectory by a novel MPPI objective function that considers the uncertainty of every action in all trajectories. We then execute the first action of the optimal trajectory. Extensive experiments in a simulated five-zone building show that CLUE only needs seven days of training data to provide comparable energy saving as the state-of-the-art MBRL method, but with less comfort violations. Our code and dataset are available at https://github.com/ryeii/CLUE.


  • Open Access

    TODOS: Thermal sensOr Data-driven Occupancy Estimation System for Smart Buildings

    by Hamid Rajabi, Xianzhong Ding, Wan Du, Alberto Cerpa

    Occupancy sensing and estimation in large commercial buildings has become a significant problem to be solved, with applications ranging from occupancy-based HVAC control to space planning, and security, etc. Thermal sensing is a promising technology to solve this problem, being easy to deploy in practice and allowing an actual occupancy count in a particular room without violating the data and privacy concerns. While initial strides have been made to solve this problem with thermal arrays, there are many problems that remain unsolved, including accuracy performance, overlapping of sensing areas that lead to under/over-counting, and data training requirements for different zones. In this paper, we introduce TODOS 1, a novel system for estimating occupancy in intelligent buildings. TODOS uses a low-cost, low-power thermal sensor array along with a passive infrared sensor. We introduce a novel data processing pipeline that allows us to automatically extract features from the thermal images using an artificial neural network. Through an extensive experimental evaluation2, we show that TODOS provides occupancy detection accuracy of 98% to 100% under different scenarios. In addition, it solves the issue of occupancy over/under-counting by overlapping sensing areas when using multiple thermal sensors in large rooms. This is done by treating the entire area as a single input thermal image instead of partitioning the area into multiple thermal images individually processed. Furthermore, TODOS introduces a data augmentation technique that allows the generation of training data for rooms of different sizes and shapes, without requiring specific training data from each room. Using these data, TODOS can train specifically designed neural networks optimized for any room size and shape, and achieve almost the same level of occupancy detection accuracy in rooms where experimental labeled training data is available, making it a viable solution that generalizes to the different rooms in large buildings.