Sensitivity and Specificity Can Change in Opposite Directions When New Predictive Markers Are Added to Risk Models
Authors:
- 30 December 2013
- journal article
- Published by SAGE Publications in Medical Decision Making
- Vol. 34 (4) , 513-522
Abstract
When comparing prediction models, it is essential to estimate the magnitude of change in performance rather than rely solely on statistical significance. In this paper we investigate measures that estimate change in classification performance, assuming 2-group classification based on a single risk threshold. We study the value of a new biomarker when added to a baseline risk prediction model. First, simulated data are used to investigate the change in sensitivity and specificity (ΔSe and ΔSp). Second, the influence of ΔSe and ΔSp on the net reclassification improvement (NRI; sum of ΔSe and ΔSp) and on decision-analytic measures (net benefit or relative utility) is studied. We assume normal distributions for the predictors and assume correctly specified models such that the extended model has a dominating receiver operating characteristic curve relative to the baseline model. Remarkably, we observe that even when a strong marker is added it is possible that either sensitivity (for thresholds below the event rate) or specificity (for thresholds above the event rate) decreases. In these cases, decision-analytic measures provide more modest support for improved classification than NRI, even though all measures confirm that adding the marker improved classification accuracy. Our results underscore the necessity of reporting ΔSe and ΔSp separately. When a single summary is desired, decision-analytic measures allow for a simple incorporation of the misclassification costs.Keywords
This publication has 41 references indexed in Scilit:
- Rationalizing the management of pregnancies of unknown location: temporal and external validation of a risk prediction model on 1962 pregnanciesHuman Reproduction, 2013
- Prediction model to estimate presence of coronary artery disease: retrospective pooled analysis of existing cohortsBMJ, 2012
- Reporting and Methods in Clinical Prediction Research: A Systematic ReviewPLOS Medicine, 2012
- External Validation of Diagnostic Models to Estimate the Risk of Malignancy in Adnexal MassesClinical Cancer Research, 2012
- Importance of prostate volume in the European Randomised Study of Screening for Prostate Cancer (ERSPC) risk calculators: results from the prostate biopsy collaborative groupWorld Journal of Urology, 2011
- Qualitative Assessment of the Progesterone Receptor and HER2 Improves the Nottingham Prognostic Index Up to 5 Years After Breast Cancer DiagnosisJournal of Clinical Oncology, 2010
- Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort studyThe Lancet, 2009
- Genotype Score in Addition to Common Risk Factors for Prediction of Type 2 DiabetesNew England Journal of Medicine, 2008
- General Cardiovascular Risk Profile for Use in Primary CareCirculation, 2008
- A Validated Prediction Model for All Forms of Acute Coronary SyndromeJAMA, 2004