Exploring the prognostic utility of large language models versus traditional clinical models in heart failure: a pilot study
Authors:
Open Access
- 18 November 2025
- journal article
- Published by Wolters Kluwer Health in International Journal of Surgery
- Vol. 112 (3) , 5778-5788
Abstract
Background: Large language models (LLMs) show promise in clinical decision support; however, their role in risk prediction for heart failure (HF) remains uncertain. Objective: This pilot study evaluated the prognostic performance and reproducibility of two general-purpose LLMs, ChatGPT and DeepSeek, using structured clinical data and unstructured discharge summaries, compared with a conventional clinical model. Methods: Structured data from the Zigong HF study included 473 hospitalized HF patients with 33 clinical variables predicting a 90-day composite outcome of all-cause death or rehospitalization. Discharge summaries from the Medical Information Mart for Intensive Care IV (MIMIC-IV) cohort included 2091 ICU HF patients predicting 1-year all-cause mortality. Standardized prompts were used to obtain predicted probabilities from each LLM. Model predictions were compared with logistic regression results, and reproducibility was assessed using intraclass correlation coefficients. Results: In the Zigong HF study, both LLMs showed limited discrimination (AUC 0.59 for ChatGPT, 0.56 for DeepSeek), performing below the conventional model (AUC 0.63). In the MIMIC-IV cohort, ChatGPT achieved higher discrimination (AUC 0.72) than DeepSeek (AUC 0.67, P < 0.001) and comparable performance to the clinical model (AUC 0.74, P = 0.31). Decision curve analysis showed modest benefit for ChatGPT at low-to-moderate thresholds, while DeepSeek offered minimal benefit. Repeated predictions showed significant variability for both models. Conclusions: This pilot study provides preliminary evidence that LLMs have limited predictive value for structured data but show comparable performance in text-based risk prediction. These findings suggest potential for LLMs in processing unstructured clinical information and highlight the need for validation in larger, contemporary cohorts.Keywords
This publication has 32 references indexed in Scilit:
- Benchmark evaluation of DeepSeek large language models in clinical decision-makingNature Medicine, 2025
- Multi-layered data framework for enhancing postoperative outcomes and anaesthesia management through natural language processingSLAS Technology, 2025
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- The TRIPOD-LLM reporting guideline for studies using large language modelsNature Medicine, 2025
- Hybridizing mechanistic modeling and deep learning for personalized survival prediction after immune checkpoint inhibitor immunotherapynpj Systems Biology and Applications, 2024
- Comparison of Medical Research Abstracts Written by Surgical Trainees and Senior Surgeons or Generated by Large Language ModelsJAMA Network Open, 2024
- Chatbot Reliability in Managing Thoracic Surgical Clinical ScenariosThe Annals of Thoracic Surgery, 2024
- Large language models in medicineNature Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023