From Clinical Judgment to Large Language Models: Benchmarking Predictive Approaches for Unplanned Hospital Admissions
Preprint
Abstract
Background While machine learning (ML) models show strong performance for predicting unplanned hospital visits, their clinical utility relative to physician judgment remains unclear. Large language models (LLMs) offer a promising middle ground, potentially combining algorithmic accuracy with human-interpretable reasoning. Objective To directly compare the predictive performance of physicians, structured ML models, and LLMs for forecasting 30-day emergency department (ED) visits and unplanned hospital admissions under equivalent data conditions. Methods We selected 404 cases from structured EHR data and converted them into synthetic clinical vignettes using GPT-5. Thirty-five physicians evaluated these vignettes, while CLMBR-T (a machine learning model trained on structured EHR data) was applied to the original data. Eight LLMs evaluated the same vignettes. We compared discriminative performance (AUROC, AUPRC), calibration (Brier score, Expected Calibration Error), and confidence-performance relationships across all methods. Results CLMBR-T achieved the highest discriminative performance (AUROC 0.79, 95% CI: 0.75-0.83; AUPRC 0.78, 95% CI: 0.72-0.83), followed by large LLMs (DeepSeek V3, Claude 4.1 Opus, GPT-5; AUROC 0.74). Pooled physicians performed lowest (AUROC 0.65, 95% CI: 0.59-0.70; AUPRC 0.61, 95% CI: 0.54-0.68). However, LLMs showed stronger alignment with physician reasoning (correlation r=0.51-0.65) compared to CLMBR-T (r=0.37). CLMBR-T demonstrated superior confidence calibration with significant confidence-performance correlation (r=0.21, pConclusions ML models trained on structured EHR data outperform both physicians and LLMs in predictive accuracy and confidence calibration, though LLMs achieved competitive zero-shot performance and better approximated human clinical reasoning. These findings suggest hybrid approaches combining high-performance ML screening with interpretable LLM explanations may optimize both accuracy and clinical adoption. The substantial variability in physician performance highlights limitations of benchmarking against “average” clinical judgment.Keywords
This publication has 16 references indexed in Scilit:
- Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care SystemmedRxiv, 2025
- Clinical judgment shows similar and sometimes superior discrimination compared to prognostic clinical prediction models: a systematic reviewJournal of Clinical Epidemiology, 2023
- Factors Influencing Physician Prognosis: A Scoping ReviewMDM Policy & Practice, 2022
- Odds of return: a prospective study using provider assessment to predict short-term patient return visits to the emergency departmentBMJ Open, 2021
- Man vs. Machine: Comparing Physician vs. Electronic Health Record–Based Model Predictions for 30-Day Hospital ReadmissionsJournal of General Internal Medicine, 2021
- Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studiesBMJ, 2020
- Emergency medicine physicians' ability to predict hospital admission at the time of triageThe American Journal of Emergency Medicine, 2019
- Cognitive biases associated with medical decisions: a systematic reviewBMC Medical Informatics and Decision Making, 2016
- A meta-analysis of confidence and judgment accuracy in clinical decision making.Journal of Counseling Psychology, 2015
- Extent and determinants of error in physicians' prognoses in terminally ill patients: prospective cohort studyWestern Journal of Medicine, 2000