Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility
Preprint
Abstract
This paper proposes a novel scalable type of multi-agent reinforcement learning-based coordination for distributed residential energy. Cooperating agents learn to control the flexibility offered by electric vehicles, space heating and flexible loads in a partially observable stochastic environment. In the standard independent Q-learning approach, the coordination performance of agents under partial observability drops at scale in stochastic environments. Here, the novel combination of learning from off-line convex optimisations on historical data and isolating marginal contributions to total rewards in reward signals increases stability and performance at scale. Using fixed-size Q-tables, prosumers are able to assess their marginal impact on total system objectives without sharing personal data either with each other or with a central coordinator. Case studies are used to assess the fitness of different combinations of exploration sources, reward definitions, and multi-agent learning frameworks. It is demonstrated that the proposed strategies create value at individual and system levels thanks to reductions in the costs of energy imports, losses, distribution network congestion, battery depreciation and greenhouse gas emissions.Keywords
All Related Versions
- Version 1, 2022-03-07, ArXiv
- Version 2, 2022-03-27, ArXiv
- Published version: Applied Energy, 314.
This publication has 51 references indexed in Scilit:
- Coordination of resources at the edge of the electricity grid: Systematic review and taxonomyApplied Energy, 2022
- Analytics for the Sharing Economy: Mathematics, Engineering and Business Perspectives2020
- Incentive-based demand response for smart grid with reinforcement learning and deep neural networkApplied Energy, 2019
- Reinforcement learning for demand response: A review of algorithms and modeling techniquesApplied Energy, 2019
- Some Emerging Challenges in Electricity Markets2018
- Deep transfer Q-learning with virtual leader-follower for supply-demand Stackelberg game of smart gridEnergy, 2017
- Comparison of different lead–acid battery lifetime prediction models for use in simulation of stand-alone photovoltaic systemsApplied Energy, 2014
- TECHNOLOGY ROADMAP2011
- Intelligent Agents: The Key Concepts2002
- Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents1993