Minimizing Combined Sewer Overflows with Online Model-Predictive Reinforcement Learning

This paper addresses the challenges posed by urban stormwater runoff in light of increased urbanization and climate change, which strain traditional stormwater infrastructure. It focuses on mitigating Combined Sewer Overflows (CSOs) by maximizing urban runoff storage in stormwater tunnels during Wastewater Treatment Plant (WWTP) capacity overloads. Unlike passive rule-based control, this research explores adaptive control systems that utilize weather forecasts and dynamic strategies. It introduces a novel control synthesis approach, combining Model Predictive Control (MPC) and Q-learning, to optimize CSO management based on real-time weather predictions. Simulated evaluations (using EPA-SWMM, focused on the Hvidovre stormwater tunnel in Copenhagen, Denmark) show significant improvements in CSO management: 17% over classic Q-learning and 34% over Q-learning with the \uppaal model, and 72% over Rule-Based Control (RBC). Our method, MPC Q-learning, dynamically incorporates weather conditions, outperforming other Q-learning approaches.