Developing more generalizable prediction models from pooled studies and large clustered data sets
Open Access
- 5 May 2021
- journal article
- research article
- Published by Wiley in Statistics in Medicine
- Vol. 40 (15) , 3533-3559
Abstract
Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.Keywords
Funding Information
- Directorate-General for Research and Innovation (825746)
- ZonMw (91617050)
This publication has 88 references indexed in Scilit:
- Developing and validating risk prediction models in an individual participant data meta-analysisBMC Medical Research Methodology, 2014
- Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variableBMC Medical Research Methodology, 2012
- Apixaban versus Warfarin in Patients with Atrial FibrillationNew England Journal of Medicine, 2011
- External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect CoefficientsAmerican Journal of Epidemiology, 2010
- The Performance of Risk Prediction ModelsBiometrical Journal, 2008
- A Re-Evaluation of Random-Effects Meta-AnalysisJournal of the Royal Statistical Society Series A: Statistics in Society, 2008
- Variable selection under multiple imputation using the bootstrap in a prognostic studyBMC Medical Research Methodology, 2007
- Ruling out deep venous thrombosis in primary careThrombosis and Haemostasis, 2005
- Improved tests for a random effects meta‐regression with a single covariateStatistics in Medicine, 2003
- Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data setsStatistics in Medicine, 2000