Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study
Open Access
- 17 July 2024
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 12, e55799
Abstract
Background: Large language models show promise for improving radiology workflows, but their performance on structured radiological tasks such as Reporting and Data Systems (RADS) categorization remains unexplored. Objective: This study aims to evaluate 3 large language model chatbots—Claude-2, GPT-3.5, and GPT-4—on assigning RADS categories to radiology reports and assess the impact of different prompting strategies. Methods: This cross-sectional study compared 3 chatbots using 30 radiology reports (10 per RADS criteria), using a 3-level prompting strategy: zero-shot, few-shot, and guideline PDF-informed prompts. The cases were grounded in Liver Imaging Reporting & Data System (LI-RADS) version 2018, Lung CT (computed tomography) Screening Reporting & Data System (Lung-RADS) version 2022, and Ovarian-Adnexal Reporting & Data System (O-RADS) magnetic resonance imaging, meticulously prepared by board-certified radiologists. Each report underwent 6 assessments. Two blinded reviewers assessed the chatbots’ response at patient-level RADS categorization and overall ratings. The agreement across repetitions was assessed using Fleiss κ. Results: Claude-2 achieved the highest accuracy in overall ratings with few-shot prompts and guideline PDFs (prompt-2), attaining 57% (17/30) average accuracy over 6 runs and 50% (15/30) accuracy with k-pass voting. Without prompt engineering, all chatbots performed poorly. The introduction of a structured exemplar prompt (prompt-1) increased the accuracy of overall ratings for all chatbots. Providing prompt-2 further improved Claude-2’s performance, an enhancement not replicated by GPT-4. The interrun agreement was substantial for Claude-2 (k=0.66 for overall rating and k=0.69 for RADS categorization), fair for GPT-4 (k=0.39 for both), and fair for GPT-3.5 (k=0.21 for overall rating and k=0.39 for RADS categorization). All chatbots showed significantly higher accuracy with LI-RADS version 2018 than with Lung-RADS version 2022 and O-RADS (P<.05); with prompt-2, Claude-2 achieved the highest overall rating accuracy of 75% (45/60) in LI-RADS version 2018. Conclusions: When equipped with structured prompts and guideline PDFs, Claude-2 demonstrated potential in assigning RADS categories to radiology cases according to established criteria such as LI-RADS version 2018. However, the current generation of chatbots lags in accurately categorizing cases based on more recent RADS criteria.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- O-RADS US v2022: An Update from the American College of Radiology’s Ovarian-Adnexal Reporting and Data System US CommitteeRadiology, 2023
- Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging PilotJournal of the American College of Radiology, 2023
- GPT-4 for Automated Determination of Radiologic Study and Protocol Based on Radiology Request Forms: A Feasibility StudyRadiology, 2023
- Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and LimitationsRadiology, 2023
- How Chatbots and Large Language Model Artificial Intelligence Systems Will Reshape Modern MedicineJAMA Internal Medicine, 2023
- Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility StudyRadiology, 2023
- O-RADS MRI Risk Stratification System: Guide for Assessing Adnexal Lesions from the ACR O-RADS CommitteeRadiology, 2022
- State of the Art: LI-RADS for Contrast-enhanced USRadiology, 2019
- Liver Imaging Reporting and Data System (LI-RADS) Version 2018: Imaging of Hepatocellular Carcinoma in At-Risk PatientsRadiology, 2018
- LI‐RADS (Liver Imaging Reporting and Data System): Summary, discussion, and consensus of the LI‐RADS Management Working Group and future directionsHepatology, 2014