Large language models approach clinician performance in ESC cardiovascular risk stratification: a vignette-based benchmark study
Open Access
- 18 May 2026
- journal article
- Published by Oxford University Press (OUP) in European Heart Journal - Digital Health
- Vol. 7 (5)
Abstract
Aims: Guideline-based cardiovascular risk stratification requires three distinct competencies: extracting risk factor data from clinical text, computing a validated risk score, and applying guideline-defined thresholds to assign a final risk category. We evaluated contemporary large language models (LLMs) on each of these tasks within the European Society of Cardiology (ESC) SCORE2 framework and compared LLM performance against a pooled individual clinician benchmark to contextualize findings against real-world human reproducibility. Methods and results: Eleven LLMs were evaluated using 30 simulated outpatient clinical vignettes presented in both Portuguese and English. For each vignette, models extracted cardiovascular risk factors, determined SCORE2 applicability, generated 10-year risk estimates where appropriate, and assigned a final three-class ESC risk category. A committee of three cardiologists established the reference standard; eight independent clinicians provided an individual-level human benchmark. Traditional risk-factor extraction was near-perfect across all models (micro-F1 0.97–0.99). Agreement with expert-assigned final risk categories was moderate and variable (best: GPT-4o, quadratic-weighted κw 0.69, 95% CI 0.44–0.84), with 10 of 11 models more often underestimating than overestimating risk. To isolate the source of classification error, post hoc deterministic recalculation of SCORE2 was performed using model-extracted variables in eligible vignettes; this markedly improved agreement across all models (κw 0.85–0.90), demonstrating that extraction was largely intact and computational execution was the primary failure mode. The pooled individual clinician benchmark showed moderate agreement with the reference standard (κw 0.52, 95% CI 0.28–0.67), indicating that the best-performing LLMs matched or exceeded the average individual clinician on this guideline-based task. Performance was broadly consistent across Portuguese and English. Conclusion: Contemporary LLMs reliably extract cardiovascular risk information from clinical text, and the best-performing systems achieved agreement within the range of average individual clinicians on this structured task. Their principal limitation lies in downstream computation and rule application.Keywords
Funding Information
- Fundação para a Ciência e a Tecnologia (UID/50021/2025)
- Fundação para a Ciência e a Tecnologia (UID/PRR/50021/2025)
- CRAI (C628696807-00454142)
This publication has 36 references indexed in Scilit:
- Heart disease risk factors detection from electronic health records using advanced NLP and deep learning techniquesScientific Reports, 2023
- ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?European Heart Journal - Digital Health, 2023
- Addressing the Global Burden of Cardiovascular Diseases; Need for Scalable and Sustainable FrameworksGlobal Heart, 2022
- Enhanced character-level deep convolutional neural networks for cardiovascular disease predictionBMC Medical Informatics and Decision Making, 2020
- Modifiable risk factors, cardiovascular disease, and mortality in 155 722 individuals from 21 high-income, middle-income, and low-income countries (PURE): a prospective cohort studyThe Lancet, 2020
- Primary prevention of cardiovascular disease: global cardiovascular risk assessment and management in clinical practiceEuropean Heart Journal - Quality of Care and Clinical Outcomes, 2015
- Perceptions of risk: understanding cardiovascular diseaseRisk Management and Healthcare Policy, 2010
- Electronic Health Record-Based Cardiac Risk Assessment and Identification of Unmet Preventive NeedsMedical Care, 2009
- Physicians’ attitudes and adherence to use of risk scores for primary prevention of cardiovascular disease: cross-sectional survey in three world regionsCurrent Medical Research and Opinion, 2009
- Factors impeding the implementation of cardiovascular prevention guidelines: findings from a survey conducted by the European Society of CardiologyEuropean Journal of Preventive Cardiology, 2006