Traditional Machine Learning, Deep Learning, and BERT (Large Language Model) Approaches for Predicting Hospitalizations From Nurse Triage Notes: Comparative Evaluation of Resource Management
Authors:
- Dhavalkumar PatelDPDhavalkumar Patel
- Prem TimsinaPTPrem Timsina
- Larisa GorensteinLGLarisa Gorenstein
- Benjamin S GlicksbergBGBenjamin S Glicksberg
- Ganesh RautGanesh Raut
- Satya Narayan CheetiralaSCSatya Narayan Cheetirala
- Fabio SantanaFSFabio Santana
- Jules TamegueJTJules Tamegue
- Arash KiaArash Kia
- Eyal ZimlichmanEZEyal Zimlichman
- Matthew A LevinMLMatthew A Levin
- Robert FreemanRFRobert Freeman
Open Access
- 27 August 2024
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 3 (1) , e52190
Abstract
Background: Predicting hospitalization from nurse triage notes has the potential to augment care. However, there needs to be careful considerations for which models to choose for this goal. Specifically, health systems will have varying degrees of computational infrastructure available and budget constraints. Objective: To this end, we compared the performance of the deep learning, Bidirectional Encoder Representations from Transformers (BERT)–based model, Bio-Clinical-BERT, with a bag-of-words (BOW) logistic regression (LR) model incorporating term frequency–inverse document frequency (TF-IDF). These choices represent different levels of computational requirements. Methods: A retrospective analysis was conducted using data from 1,391,988 patients who visited emergency departments in the Mount Sinai Health System spanning from 2017 to 2022. The models were trained on 4 hospitals’ data and externally validated on a fifth hospital’s data. Results: The Bio-Clinical-BERT model achieved higher areas under the receiver operating characteristic curve (0.82, 0.84, and 0.85) compared to the BOW-LR-TF-IDF model (0.81, 0.83, and 0.84) across training sets of 10,000; 100,000; and ~1,000,000 patients, respectively. Notably, both models proved effective at using triage notes for prediction, despite the modest performance gap. Conclusions: Our findings suggest that simpler machine learning models such as BOW-LR-TF-IDF could serve adequately in resource-limited settings. Given the potential implications for patient care and hospital resource management, further exploration of alternative models and techniques is warranted to enhance predictive performance in this critical domain. International Registered Report Identifier (IRRID): RR2-10.1101/2023.08.07.23293699Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Evaluating the ChatGPT family of models for biomedical reasoning and classificationJournal of the American Medical Informatics Association, 2024
- Natural Language Processing for Imaging Protocol Assignment: Machine Learning for Multiclass Classification of Abdominal CT Protocols Using Indication Text DataJournal of Digital Imaging, 2022
- Comparison of state-of-the-art machine and deep learning algorithms to classify proximal humeral fractures using radiology textEuropean Journal of Radiology, 2022
- BERT for the Processing of Radiological Reports: An Attention-based Natural Language Processing AlgorithmAcademic Radiology, 2022
- Does the magic of BERT apply to medical code assignment? A quantitative studyComputers in Biology and Medicine, 2021
- A benchmark study of machine learning models for online fake news detectionMachine Learning with Applications, 2021
- Predicting adult neuroscience intensive care unit admission from emergency department triage using a retrospective, tabular-free text machine learning approachScientific Reports, 2021
- Promoting head CT exams in the emergency department triage using a machine learning modelNeuroradiology, 2019
- Prediction of emergency department patient disposition based on natural language processing of triage notesInternational Journal of Medical Informatics, 2019
- Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit*Critical Care Medicine, 2007