Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
Open Access
- 31 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 123-133
Abstract
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.Keywords
All Related Versions
Funding Information
- Agentschap Innoveren & Ondernemen (VLAIO) (InduFlexControl-2)
- FOD Economy (FlexMyHeat)
This publication has 28 references indexed in Scilit:
- An optimal solutions-guided deep reinforcement learning approach for online energy storage controlApplied Energy, 2024
- Strategic Implicit Balancing With Energy Storage Systems via Stochastic Model Predictive ControlIEEE Transactions on Energy Markets, Policy and Regulation, 2023
- Day-ahead dispatch of novel battery charging and swapping station based on distributionally robust optimizationJournal of Energy Storage, 2023
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- A market framework for grid balancing support through imbalances tradingRenewable and Sustainable Energy Reviews, 2021
- Optimal Control Strategies for Seasonal Thermal Energy Storage Systems With Market InteractionIEEE Transactions on Control Systems Technology, 2020
- Imbalance Pricing Methodology in Belgium: Implications for Industrial Consumers2020
- Very-Short-Term Probabilistic Forecasting for a Risk-Aware Participation in the Single Price Imbalance SettlementIEEE Transactions on Power Systems, 2019
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks2016