Cliff C. Kerr

We quantitatively evaluated LLMs with and without domain-specific tools (MCP servers and skills) on a Starsim disease modeling benchmark, finding that custom tools improved performance.

Mathematical modelling with agent-based models (ABMs) has gained popularity during the COVID-19 pandemic, but their complexity makes efficient and robust calibration to data challenging, particularly when applying Bayesian methods to quantify parameter uncertainty. We propose a method for calibrating ABMs that combines a Machine-Learning step with Approximate Bayesian Computation (ML-ABC). We showcase ML-ABC application with a proof-of-principle case study, in which we calibrate the Covasim -a stochastic ABM that has been used to model the English COVID-19 epidemic and inform policy at important junctions. Benchmarking against traditional Rejection-ABC (R-ABC), we illustrate the advantage of ML-ABC application in calibrating Covasim to data on hospitalisations and deaths from COVID-19 during the first and the second COVID-19 epidemic waves of 2020 and early 2021. Across scenarios, we demonstrate that using an ML screening step allows us to derive identical posterior distributions of the calibrated Covasim parameters as with the traditional R-ABC method, but faster. Specifically, we derive posterior distributions for input parameters around 52% faster when calibrating to the first epidemic wave and around 33% faster when calibrating parameters for the second epidemic wave, compared to the traditional R-ABC. Policy modelling requires calibration which is both efficient to adapt to fast-changing pandemic environments and robust to ensure confidence in policy decisions. However, existing ABM calibration often relies on myopic non-exhaustive searches in order to remain tractible, resulting in point parameter estimates. In this preliminary study, ML-ABC strictly improves upon existing ABC calibration approaches in all tested scenarios, indicating its potential to make ABC competitive with point-estimate calibration approaches. This novel approach offers a pathway to effectively calibrate ABMs in a way which is both efficient and quantifies parameter uncertainty, crucial for realising the potential of ABMs for timely and responsively modelling during an emerging epidemic.

Sciris aims to streamline the development of scientific Python code by making it easier to perform common tasks. This example illustrates how the same block of fairly typical scientific Python code -- which performs tasks like collecting data from a function running in parallel, saving and loading files, and 3D plotting -- looks like when written in 'vanilla Python' compared to using Sciris.

Sciris aims to streamline the development of scientific software by making it easier to perform common tasks. Sciris provides classes and functions that simplify access to frequently used low-level functionality in the core libraries of the scientific Python ecosystem (such as numpy for math and matplotlib for plotting), as well as in libraries of broader scope (such as multiprocess for parallelization and pickle for saving and loading objects). While low-level functionality is valuable for developing robust software applications, it can divert focus from the scientific problems being solved. Some of Sciris' key features include: ensuring consistent dictionary, list, and array types (e.g., enabling users to provide inputs as either lists or arrays); enabling ordered dictionary elements to be referenced by index; simplifying datetime arithmetic by allowing date input in multiple formats, including strings; simplifying the saving and loading of files and complex objects; and simplifying the parallel execution of code. With Sciris, users can often achieve the same functionality with fewer lines of code, avoid reinventing the wheel, and spend less time looking up recipes on Stack Overflow. This can make writing scientific code in Python faster, more pleasant, and more accessible, especially for people without extensive training in software development.

Policymakers must make management decisions despite incomplete knowledge and conflicting model projections. Little guidance exists for the rapid, representative, and unbiased collection of policy-relevant scientific input from independent modeling teams. Integrating approaches from decision analysis, expert judgment, and model aggregation, we convened multiple modeling teams to evaluate COVID-19 reopening strategies for a mid-sized United States county early in the pandemic. Projections from seventeen distinct models were inconsistent in magnitude but highly consistent in ranking interventions. The 6-mo-ahead aggregate projections were well in line with observed outbreaks in mid-sized US counties. The aggregate results showed that up to half the population could be infected with full workplace reopening, while workplace restrictions reduced median cumulative infections by 82%. Rankings of interventions were consistent across public health objectives, but there was a strong trade-off between public health outcomes and duration of workplace closures, and no win-win intermediate reopening strategies were identified. Between-model variation was high; the aggregate results thus provide valuable risk quantification for decision making. This approach can be applied to the evaluation of management interventions in any setting where models are used to inform decision making. This case study demonstrated the utility of our approach and was one of several multimodel efforts that laid the groundwork for the COVID-19 Scenario Modeling Hub, which has provided multiple rounds of real-time scenario projections for situational awareness and decision making to the Centers for Disease Control and Prevention since December 2020.

Background: The emergence of the Brazilian variant of concern, Gamma lineage (P.1), impacted the epidemiological profile of COVID-19 cases due to its higher transmissibility rate and immune evasion ability. Methods: We sequenced 305 SARS-CoV-2 whole-genomes and performed phylogenetic analyses to identify introduction events and the circulating lineages. Additionally, we use epidemiological data of COVID-19 cases, severe cases, and deaths to measure the impact of vaccination coverage and mortality risk. Results: Here we show that Gamma introduction in São José do Rio Preto, São Paulo, Brazil, was followed by the displacement of seven circulating SARS-CoV-2 variants and a rapid increase in prevalence two months after its first detection in January 2021. Moreover, Gamma variant is associated with increased mortality risk and severity of COVID-19 cases in younger age groups, which corresponds to the unvaccinated population at the time. Conclusions: Our findings highlight the beneficial effects of vaccination indicated by a pronounced reduction of severe cases and deaths in immunized individuals, reinforcing the need for rapid and massive vaccination.

This paper uses Covasim, an agent-based model (ABM) of COVID-19, to evaluate and scenarios of epidemic spread in New York State (USA) and the UK. Epidemiological parameters such as contagiousness (virus transmission rate), initial number of infected people, and probability of being tested depend on the region's demographic and geographical features, the containment measures introduced; they are calibrated to data about COVID-19 spread in the region of interest. At the first stage of our study, epidemiological data (numbers of people tested, diagnoses, critical cases, hospitalizations, and deaths) for each of the mentioned regions were analyzed. The data were characterized in terms of seasonality, stationarity, and dependency spaces, and were extrapolated using machine learning techniques to specify unknown epidemiological parameters of the model. At the second stage, the Optuna optimizer based on the tree Parzen estimation method for objective function minimization was applied to determine the model's unknown parameters. The model was validated with the historical data of 2020. The modeled results of COVID-19 spread in New York State and the UK have demonstrated that if the level of testing and containment measures is preserved, the number of positive cases in New York State remain the same during March of 2021, while in the UK it will reduce.