Reinforcement-Learning-Enabled Supervisory Control of a Full-Scale Anaerobic Digestion Facility Using a Calibrated Digital Twin
Open Access
- 24 September 2026
- journal article
- Published by American Chemical Society (ACS) in Environmental Science & Technology
- Vol. 60 (39) , 28064-28078
Abstract
Anaerobic digestion (AD) is a cornerstone of energy recovery and decarbonization at wastewater resource recovery facilities (WRRFs), yet full-scale digesters are often operated conservatively due to delayed process feedback, nonlinear stability constraints, and limited observability. This study presents a novel reinforcement learning (RL)-enabled supervisory control framework for full-scale AD that integrates a calibrated mechanistic digital twin model (DTM) with offline policy learning and counterfactual uncertainty evaluation. Specifically, the DTM was calibrated to reproduce control-relevant digester behavior. Using 2.7 years of simulated operation, a batch-constrained Q (BCQ)-reinforcement learning policy was trained to maximize biogas production while penalizing instability associated with pH excursions and elevated volatile fatty acids (VFAs)/alkalinity (ALK) ratios. Counterfactual evaluation against historical, greedy, and heuristic controls used 1 year unseen DTM simulation and 1 year of full-scale operational data. The RL policy achieved comparable mean daily biogas production (+0.35%) in simulated testing. Under the 1 year operational data, it achieved plant-equivalent mean production of 9802 m3/day, corresponding to a modest potential cumulative improvement (+2.7%, 93 405 m3/year) subject to substantial uncertainty P(Δ > 0) = 0.524. These results indicate that offline-trained RL, combined with a mechanistic digital twin and counterfactual evaluation, provides a feasible and interpretable framework for developing and assessing supervisory AD control without live-plant experimentation while showing that the resulting benefit must be interpreted under explicit model-plant uncertainty.Keywords
Funding Information
- Renewable Bioproducts Institute
This publication has 37 references indexed in Scilit:
- From process fundamentals to engineering perspectives: A technical review on in-situ biogas upgrading via the hydrogenotrophic methanogenesis pathwayBiomass and Bioenergy, 2025
- Automated data transfer for digital twin applications: Two case studiesWater Environment Research, 2024
- Machine learning for high solid anaerobic digestion: Performance prediction and optimizationBioresource Technology, 2024
- Two-stage anaerobic membrane bioreactor for co-treatment of food waste and kitchen wastewater for biogas production and nutrients recoveryChemosphere, 2022
- Application of machine learning in anaerobic digestion: Perspectives and challengesBioresource Technology, 2022
- Biochemical methane potential prediction for mixed feedstocks of straw and manure in anaerobic co-digestionBioresource Technology, 2021
- Analysis of the economic viability of the use of biogas produced in wastewater treatment plants to generate electrical energyEnvironment, Development and Sustainability, 2020
- Prediction of anaerobic digestion performance and identification of critical operational parameters using machine learning algorithmsBioresource Technology, 2020
- Biogas production from anaerobic co-digestion of waste activated sludge: co-substrates and influencing parametersReviews in Environmental Science and Bio/Technology, 2019
- Anaerobic co-digestion: A critical review of mathematical modelling for performance optimizationBioresource Technology, 2016