Clara Bay

Infectious disease forecasting supports public health preparedness and response by informing resource allocation, intervention planning, and situational awareness. Infectious disease forecasting capabilities in the United States have grown over the past decade yet whether forecast quality has improved over this time remains unclear. Current evaluation metrics describe relative performance for a common set of forecast targets but cannot evaluate performance across different outcomes, seasons, and pathogens. We addressed this challenge by developing a novel skill score metric to assess forecasts based on performance relative to two benchmark models: a hindcast as an idealized upper bound of performance and a baseline model as a minimal performance standard. To study the impact of the baseline choice, we considered three types of baseline models with varying levels of complexity. Using this skill score framework, we retrospectively evaluated short-term influenza (influenza-like illness and hospitalization) and COVID-19 (case, death, and hospitalization) forecasts, focusing on how the performance of team and multi-model ensemble forecasts changed over time. The ensemble outperformed 99% of forecasts from a simplistic, naïve baseline, and 84-91% of forecasts from more complex baseline models that incorporate recent data. We found consistent improvements in ensemble and team forecast skill over time for both influenza forecast targets, with 4-week horizon forecasts in the most recent year having higher skill scores than 1-week horizon forecasts in the first year of each challenge. Results were mixed for COVID-19 forecasts. This framework provides a comprehensive picture of progress in infectious disease forecasting over the past decade enabled by improved standardization, data, models, and collaboration.

Forecasting influenza hospitalizations informs public health preparedness, yet questions remain about which types of forecasts best guide action. We evaluate categorical trend forecasts, which communicate probabilities of upcoming increases or decreases in epidemic trajectories, submitted to CDC’s FluSight Forecasting Challenge between Fall-2024 and Spring-2026. Teams submitted probability distributions over five categories describing direction and magnitude of week-over-week changes in laboratory-confirmed influenza hospital admissions. We assessed performance using Ranked Probability Skill Score, Brier Skill Score, and measures of forecast-observation agreement. Most models outperformed an equal-probability baseline; the FluSight ensemble ranked among the top three in the 2024–25 and 2025–26 seasons. Forecasts were most accurate during stable periods and least during periods of rapid change, with most models underestimating observed trends. Conclusions were robust to choice of scoring metric and reference model. These results support categorical trend ensembles as an approach to communicating infectious disease forecasts that may inform public health decision-making.

In recent years, the use of multi-model ensemble projections in infectious disease modeling has become an established methodological approach to account for and integrate across uncertainties and structural differences present in individual models. However, the creation of long-term ensemble projections through these coordinated efforts is resource-intensive, demanding the input of multiple research teams and substantial computational power. This typically limits the ability to refine projections, update the selection of plausible epidemic trajectories, or expand the number of scenarios that can be assessed, even as new empirical data become available. To address this challenge, we define an adaptive ensemble approach that, analogously to a multi-model particle filtering method, dynamically selects individual model trajectories based on observed data throughout the epidemic projection period. We demonstrate the effectiveness of this methodology using the U.S. Flu Scenario Modeling Hub (SMH) projections for influenza hospitalizations in the United States during the 2023-2024 and 2024-2025 winter seasons. Our findings show that the adaptive ensemble yields improved predictive accuracy with respect to the original SMH ensemble projections across several scoring rules and geographical resolutions. Furthermore, the adaptive ensemble approach offers two additional applications: i) the dynamic assignment of posterior probabilities to epidemic scenarios, identifying the most plausible scenario, and representing how reality is captured by a combination of scenarios, and ii) the potential use for short-term forecasting. The adaptive ensemble approach is able to identify the most likely scenarios for the 2023-2024 and 2024-2025 U.S. influenza seasons, even in the early stages of the epidemic. It outperforms, retrospectively, a baseline model in short-term forecasting of influenza hospitalizations in the United States during the two seasons across various horizons and scoring rules, showing potential to contribute to real-time collaborative forecasting challenges such as CDC’s FluSight. The proposed approach offers an efficient or low-resource strategy to increase the impact of multi-model epidemic projections by providing real-time support to modeling teams, public health authorities, and decision-makers.

Six years after its emergence, SARS-CoV-2 continues to have a substantial burden, however, the impact of vaccination and the optimal timing of its rollout remain uncertain. To explore these uncertainties, the US Scenario Modeling Hub convened its 19th round of ensemble projections for COVID-19 hospitalizations and deaths in the United States. Eight teams provided outcomes for each US state and nationally from April 2025 to April 2026 under five scenarios regarding vaccine recommendations and timing. We assessed recommendations with two eligibility scenarios (high-risk individuals only and all-eligible) and two timing scenarios (classic start: mid-August, earlier start: late June). These were crossed to create four scenarios and were compared against a counterfactual scenario with no vaccination. We found that compared to no vaccination, our ensemble projections estimated 90,000 (95% PI 53,000-126,000) hospitalizations averted in the high-risk and classic timing scenario across the US. Expanding coverage averted an additional 26,000 (95% PI 14,000-39,000) hospitalizations, which when coupled with earlier vaccination timing further reduced national hospitalizations by 15,000 (95% PI −3,000-33,000). These findings estimate significant benefits from a broad all-eligible vaccination recommendation, and suggest an additional benefit is likely to be gained from an earlier vaccination campaign.

Scoring rules are critical for evaluating the predictive performance of epidemic models by quantifying how well their projections and forecasts align with observed data. In this study, we introduce the energy score as a robust performance metric for stochastic trajectory-based epidemic models. As a multivariate extension of the continuous ranked probability score (CRPS), the energy score provides a single, unified measure for time-series predictions. It evaluates both calibration and sharpness by considering the distances between individual trajectories and observed data, as well as the inter-trajectory variability. We provide an overview of how the energy score can be applied to assess both scenario projections and forecasts in this format, with a particular focus on a detailed analysis of the Scenario Modeling Hub results for the 2023-2024 influenza season. By comparing the energy score to the widely used weighted interval score (WIS), we demonstrate its utility as a powerful tool for evaluating epidemic models, especially in scenarios requiring integration of predictions across multiple target outcomes into a single, interpretable metric. Author summary Epidemic model predictions are often evaluated using scoring rules, such as the weighted interval score (WIS), which require outputs in interval or quantile formats. However, epidemic models often produce outputs as collections of stochastic trajectories, which are then summarized into quantiles for evaluation. In this study, we introduce the energy score as a scoring metric specifically designed for evaluating stochastic trajectories without requiring conversion to other formats. The energy score provides a rigorous assessment by accounting for both the variability among trajectories and their alignment with observed data. Using publicly available data, we demonstrate that the energy score is a reliable and effective metric for evaluating epidemic model predictions in their native stochastic trajectory format.