RWRoss D. Williams

Ross D. Williams

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ross D. Williams has a total of 162 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Evaluating large-scale propensity score adjustment when sample size is small

    by Fleur Vereijken, Jenna M Reps, Marc A Suchard, Linying Zhang, George Hripcsak, Peter R Rijnbeek, Ross D Williams, Martijn J Schuemie

    Background: Large-scale propensity score (LSPS) models are increasingly used to control confounding in observational studies, but their reliability in small-sample settings is unclear. Small samples can limit a study’s ability to estimate propensity scores accurately and achieve adequate covariate balance, raising concerns about insufficient confounding adjustment. Resulting in small datasets being excluded, despite potentially containing valid information. Methods: We evaluated the balancing and bias reduction performance of LSPS models using six target-comparator pairs. To emulate a distributed data network analysis setting, we partitioned two large real-world data sources into smaller subsets. Within each subset, LSPS models were fit locally and evaluated on covariate balance, treatment effect bias and precision after empirical calibration, using real negative controls and synthetic positive controls. Performance was compared to a global LSPS model trained on the full dataset. Results: Across most target-comparator pairs, LSPS models substantially reduced bias and, when applying empirical calibration, improved precision. PS matching was more resilient to small sample sizes than PS stratification, achieving adequate balance even at n = 500, while stratification sometimes failed at n = 4000. A slight bias increase was observed at the smallest sample sizes, though not universally. Standard balance diagnostics consistently failed below n = 20 000, while a recently proposed diagnostic accounting for chance imbalance did not. Conclusions: LSPS models generally provide reliable bias reduction in small-sample settings, supporting their use in federated analyses. However, standard balance diagnostics may be misleading in small samples, and alternatives should be considered, such as significance checking. When LSPS fails to reduce bias adequately, additional adjustment strategies are required.

  • Open Access

    Estimating the Sample Size Required to Minimize Model Uncertainty Across Different Risk Profiles

    by Fleur Vereijken, Jenna M. Reps, Egill A. Fridgeirsson, Toby Hackmann, Ewout W. Steyerberg, Peter Rijnbeek, Ross D. Williams

  • Open Access

    Loss function influence on hyperparameter optimization for observational healthcare prediction models

    by Fleur Vereijken, Jenna M Reps, Peter Rijnbeek, Ross D Williams

    Objectives: Prediction models are increasingly used in healthcare for risk stratification and personalized care. Many models are developed using machine learning, which requires tuning hyperparameters to maximize performance based on a chosen loss function metric. In healthcare, the area under the receiver operating characteristic curve (AUROC) is commonly used for this purpose, but it may not always be the most appropriate choice for every clinical application. We empirically characterize whether the choice of loss function metric in hyperparameter optimization leads to systematic differences in model behavior across several clinical prediction tasks using real-world healthcare data. Methods: We utilized fifteen different loss function metrics to guide hyperparameter selection across three clinical prediction tasks and four machine learning algorithms. We then compared how loss function metric choice affected selected hyperparameters, overall performance, and individual predicted probabilities. Results: We observed that certain hyperparameters tended to have similar optimal values across different loss function metrics, although this pattern differed by algorithm. The best-performing models, evaluated using AUROC, were often not the models with hyperparameters optimized using AUROC. While models performed similarly at a population level, based on discrimination and calibration. The choice of the loss function metric had significant impact on the individual predicted risk for a patient. Discussion: The predictive multiplicity observed can have significant impact on the patient level, while not observed in the population level model evaluation. Conclusion: Predictive multiplicity can have a serious impact on patient treatment decisions but is not yet well understood.


  • Open Access

    Characteristics, treatment and survival of patients with chondrosarcoma in five European countries: a DARWIN EU® cohort study

    by Anton Barchuk, Cesar Barboza, Julieta Politi, Berta Raventós, Peter Prinsen, Jelle Evers, Vincent Ky Ho, Michiel Aj Van de Sande, Eric Fey, Kimmo Porkka, et al.

    Background and purpose: Chondrosarcoma is a rare bone malignancy with a poor response to systemic therapy in advanced stages. European-level epidemiological data remain scarce. This study aimed to characterise patient demographics, treatments and survival using real-world data to inform regulatory decisions about the feasibility and design of new trials for the systemic treatment of chondrosarcoma. Patient/material and methods: This cohort study, part of the DARWIN EU® initiative, analysed data from six healthcare databases in Finland, France, the Netherlands, Spain and the UK. Patients diagnosed with chondrosarcoma between 2010 and 2022 were identified. Standardised analyses were performed within a federated network using the Observational Medical Outcomes Partnership (OMOP) Common Data Model. Results: A total of 2,498 chondrosarcoma patient records were identified, covering at least 2,356 unique patients. Median age at diagnosis was 52–55 years, with a balanced sex distribution. Surgical treatment was the most common intervention, recorded in 15.2% to 88.9% of patients, depending on the database. Fewer than 5% received systemic anticancer therapy, and radiotherapy was reported in fewer than 7%. The 10-year overall survival (OS) ranged from 58% (95% confidence interval [CI]: 43–78) to 80% (95% CI: 78–82), with restricted mean survival between 7.4 and 8.7 years. In the Netherlands, patients with late-stage, metastatic or high-grade disease showed significantly poorer outcomes. Interpretation: This study demonstrates the feasibility of using real-world data across Europe to describe chondrosarcoma patients. Most had early-stage, low-grade disease amenable to surgery, with limited use of systemic therapies. Survival was generally favourable, except in advanced disease. Clinical trials remain difficult due to the rarity of advanced chondrosarcoma and the lack of standards.


  • A lossless one-shot distributed algorithm for addressing heterogeneity in multi-site generalized linear models

    by Bingyu Zhang, Qiong Wu, Jenna M Reps, Lu Li, Jiayi Tong, Yiwen Lu, Dazheng Zhang, Juan Manuel Ramirez-Anguita, Jiang Bian, Milou T Brand, et al.

    Objective: We propose Heterogeneity-aware Collaborative One-shot Lossless Algorithm for Generalized Linear Model (COLA-GLM-H), a novel one-shot lossless distributed algorithm that enables the integration of heterogeneous multi-institutional data while relying solely on instituion-level summary information rather than patient-level data. Materials and Methods: Generalized Linear Models (GLMs) are widely used in medical research for analyzing diverse outcome types. In multi-institution settings, we demonstrated that the global likelihood can be reconstructed using only institution-level summary statistics, enabling lossless estimation without accessing individual records. We validated COLA-GLM-H in two real-world studies: (1) an emulated U.S. pediatric centralized network (719,383 patients) evaluating long-term cardiovascular risks following COVID-19, and (2) an internationally decentralized network of 120,429 hospitalized patients from seven databases across three countries assessing risk factors for COVID-19 mortality. Results: In the centralized network, COLA-GLM-H produced estimates identical to those from pooled analyses. In the decentralized setting, the algorithm effectively integrated heterogeneous data across multiple clinical institutions using a single communication round. Conclusions: COLA-GLM-H provides a lossless, communication-efficient, and computation-efficient solution for multi-institutional research using only institution-level summary data. It accounts for between-institution heterogeneity and supports all outcome types within the exponential family, enabling secure, scalable, and accurate analysis in collaborative clinical research.


  • Open Access

    Longitudinal trajectories of polypharmacy in older people, and their association with the risk of mortality: a joint latent class model analysis of real-world data from the UK and the Netherlands

    by Leena Elhussein, Ross D Williams, Wai Yi Man, Edward Burn, Antonella Delmestri, Victoria Y Strauss, Daniel Prieto-Alhambra

    Objective: Polypharmacy is the use of multiple drugs. Many definitions have been established for polypharmacy, often cross-sectionally, despite it naturally changing over time. In this study, we aimed to identify clusters of older people with distinct polypharmacy trajectories over time and associated mortality risks. We then characterised the identified clusters and assessed their generalisability in two external databases. Methods: Data were extracted from three primary care databases: the UK Clinical Practice Research Datalink (CPRD) GOLD, CPRD Aurum and the Dutch Integrated Primary Care Information (IPCI). People aged ≥65 on 1 January 2015 were included. Polypharmacy, defined as the cumulative number of prescribed ingredients, was calculated at baseline and at the end of each subsequent follow-up year (2015–19). We applied joint latent class modelling, which divides the population into clusters with different trajectories and associated mortality risks. The model was trained in GOLD and validated in Aurum and IPCI. Results: Four clusters were identified and characterised based on polypharmacy baseline and rate of progression: low-steady, intermediate-slow/increasing, intermediate-fast/increasing and high-decreasing. The high-decreasing cluster had the highest average baseline polypharmacy (intercept = 23.4) and prevalence of non-cancer chronic comorbidities, whilst the intermediate-fast/increasing had the steepest polypharmacy rate of progression per year (slope = 6.4), highest baseline and cumulative incidence of cancer, and worst survival outcome. Good validation was found in Aurum and IPCI. Conclusion: High baseline levels and increasing levels of polypharmacy were associated with an increased mortality risk in older people. The clusters identified in this study were externally validated in two European databases, confirming their robustness and generalisability.


  • Top Cited Papers
    Open Access

    Implementation and Updating of Clinical Prediction Models: A Systematic Review

    by Alexander Saelmans, Tom Seinen, Victor Pera, Aniek F. Markus, Egill Fridgeirsson, Luis H. John, Lieke Schiphof-Godart, Peter Rijnbeek, Jenna Reps, Ross Williams


  • Open Access

    Comparing penalization methods for linear models on large observational health data

    by Egill A Fridgeirsson, Ross Williams, Peter Rijnbeek, Marc A Suchard, Jenna M Reps

    Objective: This study evaluates regularization variants in logistic regression (L1, L2, ElasticNet, Adaptive L1, Adaptive ElasticNet, Broken adaptive ridge [BAR], and Iterative hard thresholding [IHT]) for discrimination and calibration performance, focusing on both internal and external validation. Materials and Methods: We use data from 5 US claims and electronic health record databases and develop models for various outcomes in a major depressive disorder patient population. We externally validate all models in the other databases. We use a train-test split of 75%/25% and evaluate performance with discrimination and calibration. Statistical analysis for difference in performance uses Friedman’s test and critical difference diagrams. Results: Of the 840 models we develop, L1 and ElasticNet emerge as superior in both internal and external discrimination, with a notable AUC difference. BAR and IHT show the best internal calibration, without a clear external calibration leader. ElasticNet typically has larger model sizes than L1. Methods like IHT and BAR, while slightly less discriminative, significantly reduce model complexity. Conclusion: L1 and ElasticNet offer the best discriminative performance in logistic regression for healthcare predictions, maintaining robustness across validations. For simpler, more interpretable models, L0-based methods (IHT and BAR) are advantageous, providing greater parsimony and calibration with fewer features. This study aids in selecting suitable regularization techniques for healthcare prediction models, balancing performance, complexity, and interpretability.


  • Open Access

    The DELPHI Library: Improving Model Validation, Transparency and Dissemination Through a Centralised Library of Prediction Models

    by Ross D. Williams, Sicco Den Otter, Jenna M. Reps, Peter R. Rijnbeek

    The Deposit, Evaluate and Lookup Predictive Healthcare Information (DELPHI) library provides a centralised location for the depositing, exploring and analysing of patient-level prediction models that are compatible with data mapped to the observational medical outcomes partnership common data model.


  • Open Access

    Using Iterative Pairwise External Validation to Contextualize Prediction Model Performance: A Use Case Predicting 1-Year Heart Failure Risk in Patients with Diabetes Across Five Data Sources

    by Ross D. Williams, Jenna M. Reps, Jan A. Kors, Patrick B. Ryan, Ewout Steyerberg, Katia M. Verhamme, Peter R. Rijnbeek

    Introduction External validation of prediction models is increasingly being seen as a minimum requirement for acceptance in clinical practice. However, the lack of interoperability of healthcare databases has been the biggest barrier to this occurring on a large scale. Recent improvements in database interoperability enable a standardized analytical framework for model development and external validation. External validation of a model in a new database lacks context, whereby the external validation can be compared with a benchmark in this database. Iterative pairwise external validation (IPEV) is a framework that uses a rotating model development and validation approach to contextualize the assessment of performance across a network of databases. As a use case, we predicted 1-year risk of heart failure in patients with type 2 diabetes mellitus. Methods The method follows a two-step process involving (1) development of baseline and data-driven models in each database according to best practices and (2) validation of these models across the remaining databases. We introduce a heatmap visualization that supports the assessment of the internal and external model performance in all available databases. As a use case, we developed and validated models to predict 1-year risk of heart failure in patients initializing a second pharmacological intervention for type 2 diabetes mellitus. We leveraged the power of the Observational Medical Outcomes Partnership common data model to create an open-source software package to increase the consistency, speed, and transparency of this process. Results A total of 403,187 patients from five databases were included in the study. We developed five models that, when assessed internally, had a discriminative performance ranging from 0.73 to 0.81 area under the receiver operating characteristic curve with acceptable calibration. When we externally validated these models in a new database, three models achieved consistent performance and in context often performed similarly to models developed in the database itself. The visualization of IPEV provided valuable insights. From this, we identified the model developed in the Commercial Claims and Encounters (CCAE) database as the best performing model overall. Conclusion Using IPEV lends weight to the model development process. The rotation of development through multiple databases provides context to model assessment, leading to improved understanding of transportability and generalizability. The inclusion of a baseline model in all modelling steps provides further context to the performance gains of increasing model complexity. The CCAE model was identified as a candidate for clinical use. The use case demonstrates that IPEV provides a huge opportunity in a new era of standardised data and analytics to improve insight into and trust in prediction models at an unprecedented scale.