Trustworthiness of a machine learning early warning model in medical and surgical inpatients
Open Access
- 26 December 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 8 (1) , ooae156
Abstract
Objectives: In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards. Materials and Methods: We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC). Results: From August 23, 2021 to March 31, 2022, 35 937 medical admissions had 2173 (6.05%) AE compared to 25 214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (P <.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively. Discussion: Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment. Conclusions: Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.Keywords
Funding Information
- Minnesota Partnership for Biotechnology and Medical Genomics
- Center for Learning Health System Sciences
- University of Minnesota
This publication has 23 references indexed in Scilit:
- Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19Annals of the American Thoracic Society, 2021
- Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodologyBMJ, 2020
- Early Detection of In-Patient Deterioration: One Prediction Model Does Not Fit AllCritical Care Explorations, 2020
- Comparison of Early Warning Scoring Systems for Hospitalized Patients With and Without Infection at Risk for In-Hospital Mortality and Transfer to the Intensive Care UnitJAMA Network Open, 2020
- Potential Biases in Machine Learning Algorithms Using Electronic Health Record DataJAMA Internal Medicine, 2018
- National Early Warning Score (NEWS) as an emergency department predictor of disease severity and 90-day survival in the acutely dyspneic patient – a prospective observational studyScandinavian Journal of Trauma, Resuscitation and Emergency Medicine, 2016
- Early Warning System Scores for Clinical Deterioration in Hospitalized Patients: A Systematic ReviewAnnals of the American Thoracic Society, 2014
- Monitoring Vital Signs: Development of a Modified Early Warning Scoring (Mews) System for General Wards in a Developing CountryPLOS ONE, 2014
- Developing predictive models using electronic medical records: challenges and pitfalls.2013
- The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and deathResuscitation, 2013