Development of an LLM pipeline exceeding physician-documented cardiovascular risk scores under routine clinical conditions

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Aims: Risk scores are essential to evidence-based cardiovascular care, but manual calculation is labour intensive and error prone. Large language models (LLMs) could automate this process, yet LLMs are limited by their propensity for calculation errors and factual hallucinations. Pipelines separating LLM-based data extraction from deterministic score computation may improve reliability and transparency. Methods and results: We conducted a retrospective diagnostic study at a quaternary heart centre in Germany (January 2020 to July 2023). Patients with atrial fibrillation (n = 179) from an ablation registry and patients with severe aortic stenosis (n = 76) evaluated by a heart team were included. Six LLMs (GPT-5.2, Gemini 3.1 Pro, DeepSeek-R1, Qwen3, GPT-OSS 120B, and Kimi K2.5) were tested in standalone, retrieval-augmented generation (RAG), and pipeline configurations to compute HAS-BLED, CHA2DS2-VASc, and EuroSCORE II scores from routine clinical reports. Accuracy was assessed against expert-adjudicated ground truth using root mean squared error (RMSE) and Krippendorff’s α to evaluate numerical deviation and categorical agreement, respectively. Pipeline-generated scores showed substantially higher agreement with expert adjudication than standalone LLMs, LLMs with RAG, and treating physicians (mean Krippendorff’s α: 0.78 vs. 0.32 vs. 0.39 vs. 0.31) and lower deviation from ground truth (mean RMSE: 0.89 vs. 5.81 vs. 1.85 vs. 1.34). Conclusion: Pipelines combining expert-curated knowledge injection, LLM-based clinical data extraction, and deterministic score calculation enable accurate and scalable cardiovascular risk score computation from unstructured real-world clinical data, outperforming physician-documented scores. Such pipelines could form the basis for clinical decision-support systems that automate routine risk assessment, reduce clinician workload, and promote more consistent evidence-based care.
Funding Information
  • Deutsches Zentrum für Herz-Kreislaufforschung
  • Bundesministerium für Forschung, Technologie und Raumfahrt
  • Charité – Universitätsmedizin Berlin
  • BIH Charité Digital Clinician Scientist Program
  • Berlin Institute of Health

This publication has 24 references indexed in Scilit: