Development and Temporal Validation of a Machine Learning Model to Predict Clinical Deterioration
Authors:
- 6 December 2023
- journal article
- research article
- Published by American Academy of Pediatrics (AAP) in Hospital Pediatrics
- Vol. 14 (1) , 11-20
Abstract
Early warning scores detecting clinical deterioration in pediatric inpatients have wide-ranging performance and use a limited number of clinical features. This study developed a machine learning model leveraging multiple static and dynamic clinical features from the electronic health record to predict the composite outcome of unplanned transfer to the ICU within 24 hours and inpatient mortality within 48 hours in hospitalized children. Using a retrospective development cohort of 17 630 encounters across 10 388 patients, 2 machine learning models (light gradient boosting machine [LGBM] and random forest) were trained on 542 features and compared with our institutional Pediatric Early Warning Score (I-PEWS). The LGBM model significantly outperformed I-PEWS based on receiver operating characteristic curve (AUROC) for the composite outcome of ICU transfer or mortality for both internal validation and temporal validation cohorts (AUROC 0.785 95% confidence interval [0.780–0.791] vs 0.708 [0.701–0.715] for temporal validation) as well as lead-time before deterioration events (median 11 hours vs 3 hours; P = .004). However, LGBM performance as evaluated by precision recall curve was lesser in the temporal validation cohort with associated decreased positive predictive value (6% vs 29%) and increased number needed to evaluate (17 vs 3) compared with I-PEWS. Our electronic health record based machine learning model demonstrated improved AUROC and lead-time in predicting clinical deterioration in pediatric inpatients 24 to 48 hours in advance compared with I-PEWS. Further work is needed to optimize model positive predictive value to allow for integration into clinical practice.Keywords
This publication has 36 references indexed in Scilit:
- Comparison of Three Acute Care Pediatric Early Warning Scoring ToolsJournal of Pediatric Nursing, 2013
- Validation of the Cardiac Children's Hospital Early Warning Score: An Early Warning Scoring Tool to Prevent Cardiopulmonary Arrests in Children with Heart DiseaseCongenital Heart Disease, 2013
- Development of a score to predict clinical deterioration in hospitalized childrenJournal of Hospital Medicine, 2011
- Multicentre validation of the bedside paediatric early warning system score: a severity of illness score to detect evolving critical illness in hospitalised childrenCritical Care, 2011
- Sensitivity of the Pediatric Early Warning Score to Identify Patient DeteriorationPediatrics, 2010
- Development and initial validation of the Bedside Paediatric Early Warning System scoreCritical Care, 2009
- Reduction of hospital mortality and of preventable cardiac arrest and death on introduction of a pediatric medical emergency team*Pediatric Critical Care Medicine, 2009
- Prospective Evaluation of a Pediatric Inpatient Early Warning Scoring SystemJournal for Specialists in Pediatric Nursing, 2009
- The pediatric early warning system score: A severity of illness score to predict urgent medical need in hospitalized childrenJournal of Critical Care, 2006
- Inpatient transfers to the intensive care unitJournal of General Internal Medicine, 2003