Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
Preprint
Abstract
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- An optimal solutions-guided deep reinforcement learning approach for online energy storage controlApplied Energy, 2024
- Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage ControlIEEE Transactions on Control of Network Systems, 2023
- Strategic Implicit Balancing With Energy Storage Systems via Stochastic Model Predictive ControlIEEE Transactions on Energy Markets, Policy and Regulation, 2023
- Day-ahead dispatch of novel battery charging and swapping station based on distributionally robust optimizationJournal of Energy Storage, 2023
- Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation ErrorsIEEE Transactions on Neural Networks and Learning Systems, 2021
- A market framework for grid balancing support through imbalances tradingRenewable and Sustainable Energy Reviews, 2021
- Optimal Control Strategies for Seasonal Thermal Energy Storage Systems With Market InteractionIEEE Transactions on Control Systems Technology, 2020
- Deep Reinforcement Learning-Based Energy Storage Arbitrage With Accurate Lithium-Ion Battery Degradation ModelIEEE Transactions on Smart Grid, 2020
- Very-Short-Term Probabilistic Forecasting for a Risk-Aware Participation in the Single Price Imbalance SettlementIEEE Transactions on Power Systems, 2019
- Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement LearningIEEE Transactions on Smart Grid, 2016