Prospective evaluation of a large language model clinical decision support system in the emergency department
- 19 August 2026
- journal article
- research article
- Published by Springer Nature in Nature Medicine
- Vol. 32 (10) , 3741-3747
Abstract
Prospective evidence for artificial intelligence (AI)-based clinical decision support in emergency departments remains limited. Here we conducted a DECIDE-AI stage 1 evaluation of SHAKED, a clinical decision support system built on multiple large language models, in a tertiary emergency department. Over 4 weeks, 1,138 patients were analyzed across two parallel units—one using SHAKED and one following routine rotations. Clinical adoption of SHAKED declined from 68% to 30%, owing to workload-sensitive disengagement (OR = 0.72 per shift hour, 95% CI 0.62 to 0.83). Physicians preferred the use of SHAKED for radiology consultations (OR = 2.98, 95% CI 1.58 to 5.63). No adverse events were detected, and expert review rated 99 of 100 sampled outputs as clinically appropriate. Emergency department length of stay did not differ between wings (4.9 h in both, P = 0.99). Intention-to-treat analysis showed a non-significant trend toward shorter consultation cycle time (−9.4 min, P = 0.077). These findings suggest that sustained clinician engagement, rather than algorithmic accuracy, may be the key barrier to effective clinical AI use in emergency departments. They inform randomized trial design but do not justify clinical deployment of AI clinical decision support at this stage. ClinicalTrials.gov identifier: NCT06902675.Keywords
This publication has 42 references indexed in Scilit:
- Artificial intelligence in emergency medicine. A systematic literature reviewInternational Journal of Medical Informatics, 2023
- Improvements to emergency department length of stay and user satisfaction after implementation of an integrated consult orderJACEP Open, 2023
- Randomized Clinical Trials of Machine Learning Interventions in Health CareJAMA Network Open, 2022
- Artificial Intelligence in Emergency Medicine: Surmountable Barriers With Revolutionary PotentialAnnals of Emergency Medicine, 2020
- The impact of the improvement in internal medicine consultation process on ED length of stayThe American Journal of Emergency Medicine, 2018
- Systematic review of the Hawthorne effect: New concepts are needed to study research participation effectsJournal of Clinical Epidemiology, 2014
- Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research AgendaAdministration and Policy in Mental Health and Mental Health Services Research, 2010
- Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samplesStatistics in Medicine, 2009
- Systematic Review of Emergency Department Crowding: Causes, Effects, and SolutionsAnnals of Emergency Medicine, 2008
- The Quality of CareJAMA, 1988