A Semantic Evaluation Framework for Medical Report Generation Using Large Language Models
Authors:
- Haider AliHAHaider Ali
- Rashadul Islam SumonRSRashadul Islam Sumon
- Abdul Rehman KhalidAKAbdul Rehman Khalid
- Kounen FathimaKFKounen Fathima
- Hee Cheol KimHee Cheol Kim
Open Access
- 1 January 2025
- journal article
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 84 (3) , 5445-5462
Abstract
Artificial intelligence is reshaping radiology by enabling automated report generation, yet evaluating the clinical accuracy and relevance of these reports is a challenging task, as traditional natural language generation metrics like BLEU and ROUGE prioritize lexical overlap over clinical relevance. To address this gap, we propose a novel semantic assessment framework for evaluating the accuracy of artificial intelligence-generated radiology reports against ground truth references. We trained 5229 image–report pairs from the Indiana University chest X-ray dataset on the R2GenRL model and generated a benchmark dataset on test data from the Indiana University chest X-ray and MIMIC-CXR datasets. These datasets were selected for their public availability, large scale, and comprehensive coverage of diverse clinical cases in chest radiography, enabling robust evaluation and comparison with prior work. Results demonstrate that the Mistral model, particularly with task-oriented prompting, achieves superior performance (up to 91.9% accuracy), surpassing other models and closely aligning with established metrics like BERTScore-F1 (88.1%) and CLIP-Score (88.7%). Statistical analyses, including paired t-tests (p < 0.01) and analysis of variance (p < 0.05), confirm significant improvements driven by structured prompting. Failure case analysis reveals limitations, such as over-reliance on lexical similarity, underscoring the need for domain-specific fine-tuning. This framework advances the evaluation of artificial intelligence-driven (AI-driven) radiology report generation, offering a robust, clinically relevant metric for assessing semantic accuracy and paving the way for more reliable automated systems in medical imaging.Keywords
This publication has 51 references indexed in Scilit:
- Advanced deep learning and large language models: Comprehensive insights for cancer detectionImage and Vision Computing, 2025
- Automated Radiology Report Generation: A Review of Recent AdvancesIEEE Reviews in Biomedical Engineering, 2024
- Quantitative Evaluation of Large Language Models to Streamline Radiology Report Impressions: A Multimodal Retrospective AnalysisRadiology, 2024
- R2GenGPT: Radiology Report Generation with frozen LLMsMeta-Radiology, 2023
- Diagnostic error and bias in the department of radiology: a pictorial essayInsights into Imaging, 2023
- Large language models in medicineNature Medicine, 2023
- Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical ImpactRadiology, 2023
- GPT-3: Its Nature, Scope, Limits, and ConsequencesMinds and Machines, 2020
- When Radiology Report Generation Meets Knowledge GraphProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Using Word2Vec to process big text data2015