Corrected ROC analysis for misclassified binary outcomes
- 28 February 2017
- journal article
- research article
- Published by Wiley in Statistics in Medicine
- Vol. 36 (13) , 2148-2160
Abstract
Creating accurate risk prediction models from Big Data resources such as Electronic Health Records (EHRs) is a critical step toward achieving precision medicine. A major challenge in developing these tools is accounting for imperfect aspects of EHR data, particularly the potential for misclassified outcomes. Misclassification, the swapping of case and control outcome labels, is well known to bias effect size estimates for regression prediction models. In this paper, we study the effect of misclassification on accuracy assessment for risk prediction models and find that it leads to bias in the area under the curve (AUC) metric from standard ROC analysis. The extent of the bias is determined by the false positive and false negative misclassification rates as well as disease prevalence. Notably, we show that simply correcting for misclassification while building the prediction model is not sufficient to remove the bias in AUC. We therefore introduce an intuitive misclassification-adjusted ROC procedure that accounts for uncertainty in observed outcomes and produces bias-corrected estimates of the true AUC. The method requires that misclassification rates are either known or can be estimated, quantities typically required for the modeling step. The computational simplicity of our method is a key advantage, making it ideal for efficiently comparing multiple prediction models on very large datasets. Finally, we apply the correction method to a hospitalization prediction model from a cohort of over 1 million patients from the Veterans Health Administrations EHR. Implementations of the ROC correction are provided for Stata and R. Published 2017. This article is a U.S. Government work and is in the public domain in the USAKeywords
Funding Information
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Office of Research and Development
- Tom Braun
- Sebastian Zöllner,
- Phillip Boonstra
- VA IIR 11-088
- VA Information Resource Center (SDR 02-237, 98-004)
This publication has 33 references indexed in Scilit:
- Genome wide association studies in presence of misclassified binary responsesBMC Genomic Data, 2013
- Mining electronic health records: towards better research applications and clinical careNature Reviews Genetics, 2012
- Uncertain outcomes: adjusting for misclassification in antimalarial efficacy studiesEpidemiology and Infection, 2010
- Sensitivity analysis for misclassification in logistic regression via likelihood methods and predictive value weightingStatistics in Medicine, 2010
- Development of Inpatient Risk Stratification Models of Acute Kidney Injury for Use in Electronic Health RecordsMedical Decision Making, 2010
- Identifiability of Models for Multiple Diagnostic Testing in the Absence of a Gold StandardBiometrics, 2009
- A General Method for Dealing with Misclassification in Regression: The Misclassification SIMEXBiometrics, 2005
- Modelling risk when binary outcomes are subject to errorStatistics in Medicine, 2004
- Comparing disease screening tests when true disease status is ascertained only for screen positivesBiostatistics, 2001
- Bias and efficiency loss due to misclassified responses in binary regressionBiometrika, 1999