Yuanqi Gao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Yuanqi Gao has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
- Nanpeng Yu
University of California, Riverside
11 shared publications - Jie Shi
Department of Electrical and Computer Engineering, University of California at Riverside, Riverside, CA, USA
4 shared publications - Wei Wang
School of Transportation, Southeast University, Nanjing, Jiangsu, 211189, China
4 shared publications - Brandon Foggo
Department of Electrical and Computer Engineering, University of California, Riverside, CA, USA
2 shared publications - Zuzhao Ye
Department of Electrical and Computer Engineering, University of California at Riverside, Riverside, CA, USA
2 shared publications - Bryan Matthew Wong
University of California, Riverside
1 shared publication
Latest Publications
- Open Access
- 11 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 100125-100136
An Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems
Unit commitment (UC) is a fundamental problem in the day-ahead electricity market, and it is critical to solve UC problems efficiently. Mathematical optimization techniques like dynamic programming, Lagrangian relaxation, and mixed-integer quadratic programming (MIQP) are commonly adopted for UC problems. However, the calculation time of these methods increases at an exponential rate with the number of generators and energy resources, which is still the main bottleneck in the industry. Recent advances in artificial intelligence have demonstrated the capability of reinforcement learning (RL) to solve UC problems. Unfortunately, the existing research on solving UC problems with RL suffers from the curse of dimensionality when the size of UC problems grows. To deal with these problems, we propose an optimization method-assisted ensemble deep reinforcement learning algorithm, where UC problems are formulated as a Markov Decision Process (MDP) and solved by multi-step deep Q-learning in an ensemble framework. The proposed algorithm establishes a candidate action set by solving tailored optimization problems to ensure relatively high performance and the satisfaction of operational constraints. Numerical studies on three test systems show that our algorithm outperforms the baseline RL algorithm in terms of computation efficiency and operation cost. By employing the output of our proposed algorithm as a warm start, the MIQP technique can achieve further reductions in operational costs. Furthermore, the proposed algorithm shows strong generalization capacity under unforeseen operational conditions.
- 11 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 100125-100136
Open AccessAn Optimization Method-Assisted Ensemble Deep Reinforcement Learning Algorithm to Solve Unit Commitment Problems
Unit commitment (UC) is a fundamental problem in the day-ahead electricity market, and it is critical to solve UC problems efficiently. Mathematical optimization techniques like dynamic programming, Lagrangian relaxation, and mixed-integer quadratic programming (MIQP) are commonly adopted for UC problems. However, the calculation time of these methods increases at an exponential rate with the number of generators and energy resources, which is still the main bottleneck in the industry. Recent advances in artificial intelligence have demonstrated the capability of reinforcement learning (RL) to solve UC problems. Unfortunately, the existing research on solving UC problems with RL suffers from the curse of dimensionality when the size of UC problems grows. To deal with these problems, we propose an optimization method-assisted ensemble deep reinforcement learning algorithm, where UC problems are formulated as a Markov Decision Process (MDP) and solved by multi-step deep Q-learning in an ensemble framework. The proposed algorithm establishes a candidate action set by solving tailored optimization problems to ensure relatively high performance and the satisfaction of operational constraints. Numerical studies on three test systems show that our algorithm outperforms the baseline RL algorithm in terms of computation efficiency and operation cost. By employing the output of our proposed algorithm as a warm start, the MIQP technique can achieve further reductions in operational costs. Furthermore, the proposed algorithm shows strong generalization capacity under unforeseen operational conditions.
- 13 September 2022
- Published by Royal Society of Chemistry (RSC) in Physical Chemistry Chemical Physics
- Vol. 24 (39) , 24012-24020
Harnessing deep reinforcement learning to construct time-dependent optimal fields for quantum control dynamics
Deep reinforcement learning can be used as an efficient artificial intelligence approach to control time-dependent quantum dynamical systems.
- 13 September 2022
- Published by Royal Society of Chemistry (RSC) in Physical Chemistry Chemical Physics
- Vol. 24 (39) , 24012-24020
Harnessing deep reinforcement learning to construct time-dependent optimal fields for quantum control dynamics
Deep reinforcement learning can be used as an efficient artificial intelligence approach to control time-dependent quantum dynamical systems.
- Top Cited Papers
- 24 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 13 (6) , 4873-4886
Multi-Agent Deep Reinforcement Learning for Voltage Control With Coordinated Active and Reactive Power Optimization
The increasing penetration of distributed renewable energy resources causes voltage fluctuations in distribution networks. The controllable active and reactive power resources such as energy storage (ES) systems and electric vehicles (EVs) in active distribution networks play an important role in mitigating the voltage excursions. This paper proposes a two-timescale hybrid voltage control strategy based on a mixed-integer optimization method and multi-agent reinforcement learning (MARL) to reduce power loss and mitigate voltage violations. In the slow-timescale, the active and reactive power optimization problem involving capacitor banks (CBs), on-load tap changers (OLTC), and ES systems is formulated as a mixed-integer second-order cone programming problem. In the fast-timescale, the reactive power of smart inverters connected to solar photovoltaic systems and active power of EVs are adjusted to mitigate short-term voltage fluctuations with a MARL algorithm. Specifically, we propose an experience augmented multi-agent actor-critic (EA-MAAC) algorithm with an attention mechanism to learn high-quality control policies. The control policies are executed online in a decentralized manner. The proposed hybrid voltage control strategy is validated on an IEEE testing distribution feeder. The numerical results show that our proposed control strategy is not only sample-efficient and robust but also effective in mitigating voltage fluctuations.
- 24 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 13 (6) , 4873-4886
Top Cited PapersMulti-Agent Deep Reinforcement Learning for Voltage Control With Coordinated Active and Reactive Power Optimization
The increasing penetration of distributed renewable energy resources causes voltage fluctuations in distribution networks. The controllable active and reactive power resources such as energy storage (ES) systems and electric vehicles (EVs) in active distribution networks play an important role in mitigating the voltage excursions. This paper proposes a two-timescale hybrid voltage control strategy based on a mixed-integer optimization method and multi-agent reinforcement learning (MARL) to reduce power loss and mitigate voltage violations. In the slow-timescale, the active and reactive power optimization problem involving capacitor banks (CBs), on-load tap changers (OLTC), and ES systems is formulated as a mixed-integer second-order cone programming problem. In the fast-timescale, the reactive power of smart inverters connected to solar photovoltaic systems and active power of EVs are adjusted to mitigate short-term voltage fluctuations with a MARL algorithm. Specifically, we propose an experience augmented multi-agent actor-critic (EA-MAAC) algorithm with an attention mechanism to learn high-quality control policies. The control policies are executed online in a decentralized manner. The proposed hybrid voltage control strategy is validated on an IEEE testing distribution feeder. The numerical results show that our proposed control strategy is not only sample-efficient and robust but also effective in mitigating voltage fluctuations.
- Top Cited Papers
- 6 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 13 (4) , 3038-3048
Learning to Operate an Electric Vehicle Charging Station Considering Vehicle-Grid Integration
The rapid adoption of electric vehicles (EVs) calls for the widespread installation of EV charging stations. To maximize the profitability of charging stations, intelligent controllers that provide both charging and electric grid services are in great need. However, it is challenging to determine the optimal charging schedule due to the uncertain arrival time and charging demands of EVs. In this paper, we propose a novel centralized allocation and decentralized execution (CADE) reinforcement learning (RL) framework to maximize the charging station’s profit. In the centralized allocation process, EVs are allocated to either the waiting or charging spots. In the decentralized execution process, each charger makes its own charging/discharging decision while learning the action-value functions from a shared replay memory. This CADE framework significantly improves the scalability and sample efficiency of the RL algorithm. Numerical results show that the proposed CADE framework is both computationally efficient and scalable, and significantly outperforms the baseline model predictive control (MPC). We also provide an in-depth analysis of the learned action-value function to explain the inner working of the reinforcement learning agent.
- 6 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 13 (4) , 3038-3048
Top Cited PapersLearning to Operate an Electric Vehicle Charging Station Considering Vehicle-Grid Integration
The rapid adoption of electric vehicles (EVs) calls for the widespread installation of EV charging stations. To maximize the profitability of charging stations, intelligent controllers that provide both charging and electric grid services are in great need. However, it is challenging to determine the optimal charging schedule due to the uncertain arrival time and charging demands of EVs. In this paper, we propose a novel centralized allocation and decentralized execution (CADE) reinforcement learning (RL) framework to maximize the charging station’s profit. In the centralized allocation process, EVs are allocated to either the waiting or charging spots. In the decentralized execution process, each charger makes its own charging/discharging decision while learning the action-value functions from a shared replay memory. This CADE framework significantly improves the scalability and sample efficiency of the RL algorithm. Numerical results show that the proposed CADE framework is both computationally efficient and scalable, and significantly outperforms the baseline model predictive control (MPC). We also provide an in-depth analysis of the learned action-value function to explain the inner working of the reinforcement learning agent.
- Open Access
- 27 December 2021
- Published by Elsevier in Systems & Control Letters
- Vol. 160, 105065
Distributed and communication-efficient solutions to linear equations with special sparse structure
- 27 December 2021
- Published by Elsevier in Systems & Control Letters
- Vol. 160, 105065
Open AccessDistributed and communication-efficient solutions to linear equations with special sparse structure
- Top Cited Papers
- 12 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 12 (4) , 3594-3604
Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks
Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem, which determines the operation schedules for voltage regulators, on-load tap changers, and capacitors. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.
- 12 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 12 (4) , 3594-3604
Top Cited PapersConsensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks
Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem, which determines the operation schedules for voltage regulators, on-load tap changers, and capacitors. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.
- Top Cited Papers
- 26 June 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 11 (6) , 5357-5369
Batch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration
Dynamic distribution network reconfiguration (DNR) algorithms perform hourly status changes of remotely controllable switches to improve distribution system performance. The problem is typically solved by physical model-based control algorithms, which not only rely on accurate network parameters but also lack scalability. To address these limitations, this paper develops a data-driven batch-constrained reinforcement learning (RL) algorithm for the dynamic DNR problem. The proposed RL algorithm learns the network reconfiguration control policy from a finite historical operational dataset without interacting with the distribution network. The numerical study results on three distribution networks show that the proposed algorithm not only outperforms state-of-the-art RL algorithms but also improves the behavior control policy, which generated the historical operational data. The proposed algorithm is also very scalable and can find a desirable network reconfiguration solution in real-time.
- 26 June 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 11 (6) , 5357-5369
Top Cited PapersBatch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration
Dynamic distribution network reconfiguration (DNR) algorithms perform hourly status changes of remotely controllable switches to improve distribution system performance. The problem is typically solved by physical model-based control algorithms, which not only rely on accurate network parameters but also lack scalability. To address these limitations, this paper develops a data-driven batch-constrained reinforcement learning (RL) algorithm for the dynamic DNR problem. The proposed RL algorithm learns the network reconfiguration control policy from a finite historical operational dataset without interacting with the distribution network. The numerical study results on three distribution networks show that the proposed algorithm not only outperforms state-of-the-art RL algorithms but also improves the behavior control policy, which generated the historical operational data. The proposed algorithm is also very scalable and can find a desirable network reconfiguration solution in real-time.
- Top Cited Papers
- 27 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 11 (4) , 3008-3018
Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems
Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.
- 27 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 11 (4) , 3008-3018
Top Cited PapersSafe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems
Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.
- 3 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 31 (10) , 4073-4083
Information Losses in Neural Classifiers From Sampling
This article considers the subject of information losses arising from the finite data sets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled data sets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. This article then uses these bounds to explain some recent experimental findings of information compression in neural networks that cannot be explained by previous work. Finally, this article shows that not only are these bounds much smaller than existing ones, but they also correspond well with experiments.
- 3 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 31 (10) , 4073-4083
Information Losses in Neural Classifiers From Sampling
This article considers the subject of information losses arising from the finite data sets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled data sets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. This article then uses these bounds to explain some recent experimental findings of information compression in neural networks that cannot be explained by previous work. Finally, this article shows that not only are these bounds much smaller than existing ones, but they also correspond well with experiments.
- 21 October 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 21 (11) , 4822-4834
Operating Electric Vehicle Fleet for Ride-Hailing Services With Reinforcement Learning
- 21 October 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 21 (11) , 4822-4834
Operating Electric Vehicle Fleet for Ride-Hailing Services With Reinforcement Learning