Does poor methodological quality of prediction modeling studies translate to poor model performance? An illustration in traumatic brain injury
Open Access
- 5 May 2022
- journal article
- research article
- Published by Springer Nature in Diagnostic and Prognostic Research
- Vol. 6 (1) , 1-12
Abstract
Background: Prediction modeling studies often have methodological limitations, which may compromise model performance in new patients and settings. We aimed to examine the relation between methodological quality of model development studies and their performance at external validation. Methods: We systematically searched for externally validated multivariable prediction models that predict functional outcome following moderate or severe traumatic brain injury. Risk of bias and applicability of development studies was assessed with the Prediction model Risk Of Bias Assessment Tool (PROBAST). Each model was rated for its presentation with sufficient detail to be used in practice. Model performance was described in terms of discrimination (AUC), and calibration. Delta AUC (dAUC) was calculated to quantify the percentage change in discrimination between development and validation for all models. Generalized estimation equations (GEE) were used to examine the relation between methodological quality and dAUC while controlling for clustering. Results: We included 54 publications, presenting ten development studies of 18 prediction models, and 52 external validation studies, including 245 unique validations. Two development studies (four models) were found to have low risk of bias (RoB). The other eight publications (14 models) showed high or unclear RoB. The median dAUC was positive in low RoB models (dAUC 8%, [IQR − 4% to 21%]) and negative in high RoB models (dAUC − 18%, [IQR − 43% to 2%]). The GEE showed a larger average negative change in discrimination for high RoB models (− 32% (95% CI: − 48 to − 15) and unclear RoB models (− 13% (95% CI: − 16 to − 10)) compared to that seen in low RoB models. Conclusion: Lower methodological quality at model development associates with poorer model performance at external validation. Our findings emphasize the importance of adherence to methodological principles and reporting guidelines in prediction modeling studies.Keywords
Funding Information
- FP7 Health (602150, 602150, 602150, 602150)
- Research Foundation Flanders
This publication has 40 references indexed in Scilit:
- Reporting and Methods in Clinical Prediction Research: A Systematic ReviewPLOS Medicine, 2012
- External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect CoefficientsAmerican Journal of Epidemiology, 2010
- Predicting Outcome after Traumatic Brain Injury: Development and International Validation of Prognostic Scores Based on Admission CharacteristicsPLOS Medicine, 2008
- A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomesJournal of Clinical Epidemiology, 2008
- Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patientsBMJ, 2008
- Predicting Outcome after Traumatic Brain Injury: Development and Validation of a Prognostic Score Based on Admission CharacteristicsJournal of Neurotrauma, 2005
- Internal validation of predictive modelsJournal of Clinical Epidemiology, 2001
- Predicting survival using simple clinical variables: a case study in traumatic brain injuryJournal of Neurology, Neurosurgery & Psychiatry, 1999
- A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter studyJAMA, 1993
- APACHE IICritical Care Medicine, 1985