Performance assessment of different machine learning approaches in predicting diabetic ketoacidosis in adults with type 1 diabetes using electronic health records data
Authors:
Open Access
- 22 January 2021
- journal article
- research article
- Published by Wiley in Pharmacoepidemiology and Drug Safety
- Vol. 30 (5) , 610-618
Abstract
Purpose: To assess the performance of different machine learning (ML) approaches in identifying risk factors for diabetic ketoacidosis (DKA) and predicting DKA.Methods: This study applied flexible ML (XGBoost, distributed random forest [DRF] and feedforward network) and conventional ML approaches (logistic regression and least absolute shrinkage and selection operator [LASSO]) to 3400 DKA cases and 11 780 controls nested in adults with type 1 diabetes identified from Optum® de‐identified Electronic Health Record dataset (2007–2018). Area under the curve (AUC), accuracy, sensitivity and specificity were computed using fivefold cross validation, and their 95% confidence intervals (CI) were established using 1000 bootstrap samples. The importance of predictors was compared across these models.Results: In the training set, XGBoost and feedforward network yielded higher AUC values (0.89 and 0.86, respectively) than logistic regression (0.83), LASSO (0.83) and DRF (0.81). However, the AUC values were similar (0.82) among these approaches in the test set (95% CI range, 0.80–0.84). While the accuracy values >0.8 and the specificity values >0.9 for all models, the sensitivity values were only 0.4. The differences in these metrics across these models were minimal in the test set. All approaches selected some known risk factors for DKA as the top 10 features. XGBoost and DRF included more laboratory measurements or vital signs compared with conventional ML approaches, while feedforward network included more social demographics.Conclusions: In our empirical study, all ML approaches demonstrated similar performance, and identified overlapping, but different, top 10 predictors. The difference in selected top predictors needs further research.Keywords
Funding Information
- Sanofi
This publication has 23 references indexed in Scilit:
- Automated Detection and Classification of Type 1 Versus Type 2 Diabetes Using Electronic Health Record DataDiabetes Care, 2013
- Diabetes in the Emergency Department and Hospital: Acute Care of Diabetes PatientsCurrent Emergency and Hospital Medicine Reports, 2013
- Machine learning and radiologyMedical Image Analysis, 2012
- Collinearity: a review of methods to deal with it and a simulation study evaluating their performanceEcography, 2012
- Risk Prediction Measures for Case-Cohort and Nested Case-Control Designs: An Application to Cardiovascular DiseaseAmerican Journal of Epidemiology, 2012
- Positive predictive value of automated database records for diabetic ketoacidosis (DKA) in children and youth exposed to antipsychotic drugs or control medications: a tennessee medicaid studyBMC Medical Research Methodology, 2011
- pROC: an open-source package for R and S+ to analyze and compare ROC curvesBMC Bioinformatics, 2011
- Collinearity diagnostics of binary logistic regression modelJournal of Interdisciplinary Mathematics, 2010
- Advantages of the nested case-control design in diagnostic researchBMC Medical Research Methodology, 2008
- An introduction to ROC analysisPattern Recognition Letters, 2006