Graph Reinforcement Learning for Network Control via Bi-Level Optimization
Authors:
- Daniele GammelliDGDaniele Gammelli
- James HarrisonJHJames Harrison
- Kaidi YangKYKaidi Yang
- Marco PavoneMPMarco Pavone
- Filipe RodriguesFRFilipe Rodrigues
- Francisco C. PereiraFPFrancisco C. Pereira
Preprint
Abstract
Optimization problems over dynamic networks have been extensively studied and widely used in the past decades to formulate numerous real-world problems. However, (1) traditional optimization-based approaches do not scale to large networks, and (2) the design of good heuristics or approximation algorithms often requires significant manual trial-and-error. In this work, we argue that data-driven strategies can automate this process and learn efficient algorithms without compromising optimality. To do so, we present network control problems through the lens of reinforcement learning and propose a graph network-based framework to handle a broad class of problems. Instead of naively computing actions over high-dimensional graph elements, e.g., edges, we propose a bi-level formulation where we (1) specify a desired next state via RL, and (2) solve a convex program to best achieve it, leading to drastically improved scalability and performance. We further highlight a collection of desirable features to system designers, investigate design decisions, and present experiments on real-world control problems showing the utility, scalability, and flexibility of our framework.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimization2023
- Deep Reinforcement Learning Approach for Capacitated Supply Chain optimization under Demand Uncertainty2019
- Network analysis of supply chain systems: A systematic review and future researchSystems Engineering, 2012
- Dynamic modeling and control of supply chain systems: A reviewComputers & Operations Research, 2008
- DCOPF-Based LMP Simulation: Algorithm, Comparison With ACOPF, and SensitivityIEEE Transactions on Power Systems, 2007
- A survey of the optimal power flow literatureIEEE Transactions on Power Systems, 1991
- Multi-stage stochastic optimization applied to energy planningMathematical Programming, 1991
- Distributed dynamic routing schemesIEEE Communications Magazine, 1990
- Reminiscences about the origins of linear programmingOperations Research Letters, 1982
- An approximate analytic model of many-to-many demand responsive transportation systemsTransportation Research, 1978