A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains
Authors:
Top Cited Papers
Open Access
- 26 December 2025
- journal article
- research article
- Published by Springer Nature in npj Digital Medicine
- Vol. 9 (1)
Abstract
Large language models (LLMs) hold promise in clinical decision support but face major challenges in safety evaluation and effectiveness validation. We developed the Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB), a multidimensional framework built on clinical expert consensus, encompassing 30 metrics covering critical areas like critical illness recognition, guideline adherence, and medication safety, with weighted consequence measures. Thirty-two specialist physicians developed and revised 2069 open-ended Q&A items aligned with these criteria, spanning 26 clinical departments to simulate real-world scenarios. Benchmark testing of six LLMs revealed moderate overall performance (average total score 57.2%, safety 54.7%, effectiveness 62.3%), with a significant 13.3% performance drop in high-risk scenarios ( p < 0.0001). Domain-specific medical LLMs showed consistent performance advantages over general-purpose models, with relatively higher top scores in safety (0.912) and effectiveness (0.861). The findings of this study not only provide a standardized metric for evaluating the clinical application of medical LLMs, facilitating comparative analyses, risk exposure identification, and improvement directions across different scenarios, but also hold the potential to promote safer and more effective deployment of large language models in healthcare environments.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoningNature Medicine, 2025
- Towards accurate differential diagnosis with large language modelsNature, 2025
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- An evaluation framework for clinical use of large language models in patient interaction tasksNature Medicine, 2025
- Aligning Large Language Models with Humans: A Comprehensive Survey of ChatGPT’s Aptitude in PharmacologyDrugs, 2024
- Evaluation and mitigation of the limitations of large language models in clinical decision-makingNature Medicine, 2024
- From text to treatment: the crucial role of validation for generative large language models in health careThe Lancet Digital Health, 2024
- Adapted large language models can outperform medical experts in clinical text summarizationNature Medicine, 2024
- Foundation models for generalist medical artificial intelligenceNature, 2023
- Interobserver reproducibility in pathologist interpretation of columnar-lined esophagusVirchows Archiv, 2015