Alexander N. Pillai

Vaccine effectiveness (VE) estimates can vary widely between years and populations, even for the same vaccine. Estimated VE is known to be sensitive to susceptible depletion and differences in pre-vaccination infection risk between vaccinated and unvaccinated populations. However, how variation in pre-vaccination risk within and between the two groups affects VE estimates over time remains unclear. This uncertainty is especially important given negative VE estimates. We investigated the difference between estimated VE and true vaccine protection considering continuous distributions of pre-vaccination infection risk under three scenarios. When the vaccinated and unvaccinated populations differ in their mean risk, estimated VE can be higher or lower than true vaccine protection. Similar patterns arise when both populations share identical means but different risk distributions. Finally, if infection-derived immunity lasts longer than vaccine protection, annual VE estimates can vary by tens of percentage points between years despite constant true vaccine protection. These theoretical results underscore that VE studies estimate contrasting risk between vaccinated and unvaccinated individuals in a particular time and place, and VE estimates can vary counterintuitively between years and populations even with constant vaccine-induced protection. Explaining variability in estimated VE thus requires a more complete understanding of populations’ distributions of infection risk.

We evaluate approaches to vaccine distribution using an agent-based model of human activity and COVID-19 transmission calibrated to detailed trends in cases, hospitalizations, deaths, seroprevalence, and vaccine breakthrough infections in Florida, USA. We compare the incremental effectiveness for four different distribution strategies at four different levels of vaccine supply, starting in late 2020 through early 2022. Our analysis indicates that the best strategy to reduce severe outcomes would be to actively target high disease-risk individuals. This was true in every scenario, although the advantage was greatest for the intermediate vaccine availability assumptions and relatively modest compared to a simple mass vaccination approach under high vaccine availability. Ring vaccination, while generally the most effective strategy for reducing infections, ultimately proved least effective at preventing deaths. We also consider using age group as a practical surrogate measure for actual disease-risk targeting; this approach also outperforms both simple mass distribution and ring vaccination. We find that quantitative effectiveness of a strategy depends on whether effectiveness is assessed after the alpha, delta, or omicron wave. However, these differences in absolute benefit for the strategies do not change the ranking of their performance at preventing severe outcomes across vaccine availability assumptions. We use our agent-based model of SARS-CoV-2 transmission to evaluate alternative vaccine distribution strategies over a range of vaccine supply scenarios. We find that strategies targeting transmission (e.g., ring vaccination) perform best in preventing infections, but targeting disease risk prevents more instances of severe outcomes. Specifically, strategies based on age, or age and comorbidities—which do not require contact tracing—resulted in the fewest hospitalizations in our model. These strategy rankings held true across all vaccine supply scenarios and were robust to the introduction of SARS-CoV-2 variants. While the quantitative results cannot be directly applied to other settings (as we used a synthetic population calibrated to the State of Florida), the rankings of strategies should be more generalizable.

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Policymakers must make management decisions despite incomplete knowledge and conflicting model projections. Little guidance exists for the rapid, representative, and unbiased collection of policy-relevant scientific input from independent modeling teams. Integrating approaches from decision analysis, expert judgment, and model aggregation, we convened multiple modeling teams to evaluate COVID-19 reopening strategies for a mid-sized United States county early in the pandemic. Projections from seventeen distinct models were inconsistent in magnitude but highly consistent in ranking interventions. The 6-mo-ahead aggregate projections were well in line with observed outbreaks in mid-sized US counties. The aggregate results showed that up to half the population could be infected with full workplace reopening, while workplace restrictions reduced median cumulative infections by 82%. Rankings of interventions were consistent across public health objectives, but there was a strong trade-off between public health outcomes and duration of workplace closures, and no win-win intermediate reopening strategies were identified. Between-model variation was high; the aggregate results thus provide valuable risk quantification for decision making. This approach can be applied to the evaluation of management interventions in any setting where models are used to inform decision making. This case study demonstrated the utility of our approach and was one of several multimodel efforts that laid the groundwork for the COVID-19 Scenario Modeling Hub, which has provided multiple rounds of real-time scenario projections for situational awareness and decision making to the Centers for Disease Control and Prevention since December 2020.

We evaluate approaches to vaccine distribution using an agent-based model of human activity and COVID-19 transmission calibrated to detailed trends in cases, hospitalizations, deaths, seroprevalence, and vaccine breakthrough infections in Florida, USA. We compare the incremental effectiveness for four different distribution strategies at four different levels of vaccine availability, reflecting different income settings’ historical COVID-19 vaccine distribution. Our analysis indicates that the best strategy to reduce severe outcomes is to actively target high disease-risk individuals. This was true in every scenario, although the advantage was greatest for the middle-income-country availability assumptions, and relatively modest compared to a simple mass vaccination approach for rapid, high levels of vaccine availability. Ring vaccination, while generally the most effective strategy for reducing infections, ultimately proved least effective at preventing deaths. We also consider using age group as a practical, surrogate measure for actual disease-risk targeting; this approach still outperforms both simple mass distribution and ring vaccination. We also find that the magnitude of strategy effectiveness depends on when assessment occurs ( e.g ., after delta vs. after omicron variants). However, these differences in absolute benefit for the strategies do not change the ranking of their performance at preventing severe outcomes across vaccine availability assumptions.

In this report, we use a detailed simulation model to assess and project the COVID-19 epidemic in Florida. The model is a data-driven, stochastic, discrete-time, agent based model with an explicit representation of people and places. Using the model, we find that the omicron variant wave in Florida is likely to cause many more infections than occurred during the delta variant wave. Due to testing limitations and often mild symptoms, however, we anticipate that omicron infections will be underreported compared to delta. We project that reported cases of COVID-19 will continue to grow significantly and peak in early January 2022, and that the number of reported COVID-19 deaths due to omicron may be 1/3 of the total caused by the delta wave.