Extended sample size calculations for evaluation of prediction models using a threshold for classification
Authors:
Open Access
- 1 July 2025
- journal article
- research article
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 25 (1) , 1-12
Abstract
When evaluating the performance of a model for individualised risk prediction, the sample size needs to be large enough to precisely estimate the performance measures of interest. Current sample size guidance is based on precisely estimating calibration, discrimination, and net benefit, which should be the first stage of calculating the minimum required sample size. However, when a clinically important threshold is used for classification, other performance measures are also often reported. We extend the previously published guidance to precisely estimate threshold-based performance measures. We have reported closed-form solutions to estimate the sample size required to target sufficiently precise estimates of accuracy, specificity, sensitivity, positive predictive value (PPV), negative predictive value (NPV), and an iterative method to estimate the sample size required to target a sufficiently precise estimate of the F1-score, in an external evaluation study of a prediction model with a binary outcome. This approach requires the user to pre-specify the target standard error and the expected value for each performance measure alongside the outcome prevalence. We describe how the sample size formulae were derived and demonstrate their use in an example. Extension to time-to-event outcomes is also considered. In our examples, the minimum sample size required was lower than that required to precisely estimate the calibration slope, and we expect this would most often be the case. Our formulae, along with corresponding Python code and updated R, Stata and Python commands (pmvalsampsize), enable researchers to calculate the minimum sample size needed to precisely estimate threshold-based performance measures in an external evaluation study. These criteria should be used alongside previously published criteria to precisely estimate the calibration, discrimination, and net-benefit.Keywords
All Related Versions
Funding Information
- Birmingham Biomedical Research Centre
- Engineering and Physical Sciences Research Council (EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1, EP/Y018516/1)
- Cancer Research UK (C49297/A27294, PRCPJT-Nov21\100021)
This publication has 24 references indexed in Scilit:
- Graphical calibration curves and the integrated calibration index (ICI) for survival modelsStatistics in Medicine, 2020
- Statistical inference for net benefit measures in biomarker validation studiesBiometrics, 2019
- Three myths about risk thresholds for prediction modelsBMC Medicine, 2019
- Sample Size Calculation Guide - Part 4: How to Calculate the Sample Size for a Diagnostic Test Accuracy Study based on Sensitivity, Specificity, and the Area Under the ROC Curve2019
- Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluationHealth Technology Assessment, 2016
- Sample size estimation in diagnostic test studies of biomedical informaticsJournal of Biomedical Informatics, 2014
- Pseudo-observations in survival analysisStatistical Methods in Medical Research, 2009
- Confidence intervals for an effect size measure based on the Mann–Whitney statistic. Part 2: asymptotic methods and evaluationStatistics in Medicine, 2005
- Sensitivity and Specificity should be De-emphasized in Diagnostic Accuracy StudiesAcademic Radiology, 2003
- Limitations of Sensitivity, Specificity, Likelihood Ratio, and Bayesʼ Theorem in Assessing Diagnostic ProbabilitiesEpidemiology, 1997