Development and Comparative Evaluation of Three Artificial Intelligence Models (NLP, LLM, JEPA) for Predicting Triage in Emergency Departments: A 7-Month Retrospective Proof-of-Concept
Preprint
Abstract
Triage errors, including undertriage and overtriage, are persistent challenges in emergency departments (EDs). With increasing patient influx and staff shortages, the integration of artificial intelligence (AI) into triage protocols has gained attention. This study compares the performance of three AI models [Natural Language Processing (NLP), Large Language Models (LLM), and Joint Embedding Predictive Architecture (JEPA)] in predicting triage outcomes against the FRENCH scale and clinical practice.We conducted a retrospective analysis of a prospectively recruited cohort gathering adult patient triage data over a 7-month period at the Roger Salengro Hospital ED (Lille, France). Three AI models were trained and validated : (1) TRIAGEMASTER (NLP), (2) URGENTIAPARSE (LLM), and (3) EMERGINET (JEPA). Data included demographic details, verbatim chief complaints, vital signs, and triage outcomes based on the FRENCH scale and GEMSA coding. The primary outcome was the concordance of AI-predicted triage level with the FRENCH gold-standard. It was assessed thanks to various indicators : F1-Score, Weighted Kappa, Spearman, MAE, RMSE. The LLM model (URGENTIAPARSE) showed higher accuracy (composite score: 2.514) compared to JEPA (EMERGINET, 0.438) and NLP (TRIAGEMASTER, -3.511), outperforming nurse triage (-4.343). Secondary analyses highlighted the effectiveness of URGENTIAPARSE in predicting hospitalization needs (GEMSA) and its robustness with structured data versus raw transcripts (either for GEMSA prediction or for FRENCH prediction). LLM architecture, through abstraction of patient representations, offers the most accurate triage predictions among tested models. Integrating AI into ED workflows could enhance patient safety and operational efficiency, though integration into clinical workflows requires addressing model limitations and ensuring ethical transparency.Keywords
All Related Versions
- Version 1, 2025-07-01, ArXiv
- Version 2, 2025-09-11, ArXiv
- Published version: , 1.
This publication has 12 references indexed in Scilit:
- Predicting emergency department admissions using a machine-learning algorithm: a proof of concept with retrospective studyBMC Emergency Medicine, 2025
- Detecting hallucinations in large language models using semantic entropyNature, 2024
- Navigating the landscape of medical triage: Unveiling the potential and challenges of large language models and beyondThe American Journal of Emergency Medicine, 2024
- Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic reviewPLOS ONE, 2022
- The prediction of hospital length of stay using unstructured dataBMC Medical Informatics and Decision Making, 2021
- A Review of Recurrent Neural Networks: LSTM Cells and Network ArchitecturesNeural Computation, 2019
- Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity IndexAnnals of Emergency Medicine, 2018
- Challenges in the validation of triage systems at emergency departmentsJournal of Clinical Epidemiology, 2010
- Triage with the French Emergency Nurses Classification in Hospital scale: reliability and validityEuropean Journal of Emergency Medicine, 2009
- Dimensionality Reduction by Learning an Invariant Mapping2006