Flooding and Overflow Mitigation Using Deep Reinforcement Learning Based on Koopman Operator of Urban Drainage Systems
Open Access
- 13 July 2022
- journal article
- research article
- Published by American Geophysical Union (AGU) in Water Resources Research
- Vol. 58 (7)
Abstract
In recent studies, deep reinforcement learning (RL) methods have been used for the real‐time control of urban drainage systems (UDSs). However, the training process of an RL agent is computationally expensive since collecting the training data requires multiple simulations of a UDS model. An effective solution to this issue is to replace the original UDS model with an emulator, a simpler model which emulates the specified output of the system. However, the adaptability and interpretability of the existing UDS emulators still require further improvements. To address these challenges, a new emulator with increased adaptability and interpretability, called Koopman emulator, was developed by using the Koopman operator and a deep learning technique in this study. Then, the Koopman emulator was used to replace the UDS model in RL to establish an emulator‐based RL framework for fast and efficient training. This framework was used to train emulator‐based RL agents for flooding and overflow mitigation in a UDS located in Eastern China. Baseline emulators were also employed for comparison. All the methods were evaluated through simulations. The following results were obtained: (a) The Koopman emulator demonstrated a better emulation performance than the baseline emulators. (b) The RL agents based on the Koopman emulator achieved a similar control effect with a faster training process and more efficient data usage compared with the RL agents based on the storm water management model. (c) The uncertainty analysis showed that the RL agents based on the Koopman emulator exhibited acceptable robustness when facing different rainfall events and input perturbations.Keywords
Funding Information
- National Natural Science Foundation of China (52170102)
This publication has 73 references indexed in Scilit:
- A general framework for Dynamic Emulation Modelling in environmental problemsEnvironmental Modelling & Software, 2012
- An efficient initialization approach of Q-learning for mobile robotsInternational Journal of Control, Automation, and Systems, 2012
- Spectral analysis of nonlinear flowsJournal of Fluid Mechanics, 2009
- Discretization of the Frobenius–Perron Operator Using a Sparse Haar Tensor Basis: The Sparse Ulam MethodSIAM Journal on Numerical Analysis, 2009
- Stabilizability of linear switching systemsNonlinear Analysis: Hybrid Systems, 2008
- Real time control of urban wastewater systems—where do we stand today?Journal of Hydrology, 2004
- Probabilistic Sensitivity Analysis of Complex Models: A Bayesian ApproachJournal of the Royal Statistical Society Series B: Statistical Methodology, 2004
- STABILITY, INERTIA, AND ROBUST STABILITY2004
- Lyapunov's first method for strongly non-linear systemsJournal of Applied Mathematics and Mechanics, 1996
- Linearization by output injection and nonlinear observersSystems & Control Letters, 1983