Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty
Top Cited Papers
Open Access
- 20 November 2023
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 14 (1) , 1-15
Abstract
Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.Keywords
All Related Versions
Funding Information
- National Science Foundation (DEB-2028301, DEB-2037885, DEB-2126278 and DEB-2220903)
- The Eberly College of Science Barbara McClintock Science Achievement Graduate Scholarship in Biology is a graduate fellowship funded through the Pennsylvania State University.
- U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (5U24GM132013, 5U24GM132013, 5U24GM132013)
- U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences
- U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences
- Council of State and Territorial Epidemiologists (Cooperative Agreement no. NU38OT000297, Cooperative Agreement no. NU38OT000297, Cooperative Agreement no. NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297, NU38OT000297)
- Virginia Department of Health (PV-BII, and COVID-19 Modeling Program VDH-21-501-0135, PV-BII, and COVID-19 Modeling Program VDH-21-501-0135, PV-BII, and COVID-19 Modeling Program VDH-21-501-0135)
This publication has 50 references indexed in Scilit:
- The Long-Term Safety, Public Health Impact, and Cost-Effectiveness of Routine Vaccination with a Recombinant, Live-Attenuated Dengue Vaccine (Dengvaxia): A Model Comparison StudyPLOS Medicine, 2016
- Trimmed Opinion Pools and the Crowd's Calibration ProblemManagement Science, 2014
- Direct and Indirect Effects of Rotavirus Vaccination: Comparing Predictions from Transmission Dynamic ModelsPLOS ONE, 2012
- Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring RulesJournal of Business & Economic Statistics, 2011
- On the Effective Number of Climate ModelsJournal of Climate, 2011
- A systematic analysis of performance measures for classification tasksInformation Processing & Management, 2009
- Chapter 4 Forecast Combinations2006
- Combining forecasts: A review and annotated bibliographyInternational Journal of Forecasting, 1989
- Group reaction time distributions and an analysis of distribution statistics.Psychological Bulletin, 1979
- The Opinion PoolThe Annals of Mathematical Statistics, 1961