Performance of machine learning versus the national early warning score for predicting patient deterioration risk: a single-site study of emergency admissions
Open Access
- 4 December 2024
- journal article
- research article
- Published by BMJ in BMJ Health & Care Informatics
- Vol. 31 (1) , e101088
Abstract
Objectives: Increasing operational pressures on emergency departments (ED) make it imperative to quickly and accurately identify patients requiring urgent clinical intervention. The widespread adoption of electronic health records (EHR) makes rich feature patient data sets more readily available. These large data stores lend themselves to use in modern machine learning (ML) models. This paper investigates the use of transformer-based models to identify critical deterioration in unplanned ED admissions, using free-text fields, such as triage notes, and tabular data, including early warning scores (EWS). Design: A retrospective ML study. Setting: A large ED in a UK university teaching hospital. Methods: We extracted rich feature sets of routine clinical data from the EHR and systematically measured the performance of tree- and transformer-based models for predicting patient mortality or admission to critical care within 24 hours of presentation to ED. We compared our proposed models to the National EWS (NEWS). Results: Models were trained on 174 393 admission records. We found that models including free-text triage notes outperform structured tabular data models, achieving an average precision of 0.92, compared with 0.75 for tree-based models and 0.12 for NEWS. Conclusions: Our findings suggests that machine learning models using free-text data have the potential to improve clinical decision-making in the ED; our techniques significantly reduce alert rate while detecting most high-risk patients missed by NEWS.Keywords
Funding Information
- Health Data Research UK (2022.0324)
- Innovate UK (10027358)
This publication has 30 references indexed in Scilit:
- Combining the Nurse Intuition Patient Deterioration Scale with the National Early Warning Score provides more Net Benefit in predicting serious adverse events: A prospective cohort study in medical, surgical, and geriatric wardsIntensive and Critical Care Nursing, 2024
- Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature reviewInternational Journal of Medical Informatics, 2023
- A systematic review of the discrimination and absolute mortality predicted by the National Early Warning Scores according to different cut-off values and prediction windowsEuropean Journal of Internal Medicine, 2022
- Potential applications and performance of machine learning techniques and algorithms in clinical practice: A systematic reviewInternational Journal of Medical Informatics, 2022
- A Gradient Boosting Machine Learning Model for Predicting Early Mortality in the Emergency Department Triage: Devising a Nine-Point Triage ScoreJournal of General Internal Medicine, 2019
- Strengths and limitations of early warning scores: A systematic review and narrative synthesisInternational Journal of Nursing Studies, 2017
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Defining and measuring completeness of electronic health records for secondary useJournal of Biomedical Informatics, 2013
- “Identifying the hospitalised patient in crisis”—A consensus conference on the afferent limb of Rapid Response SystemsResuscitation, 2010
- Use of Electronic Health Records in U.S. HospitalsNew England Journal of Medicine, 2009