Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis
Preprint
Abstract
Objective: To compare LLM and prompting strategies for labeling Specific Incidental Findings (SIFs) occurring during lung cancer screening, in radiology reports. Materials and Methods: In this retrospective study, 300 CT reports from a Dutch academic hospital were included between 19 th March 2003 and 28 th December 2020. GPT-5, GPT-4-Turbo, Llama-3.1-8B-INST, MedGemma-4B-IT, a keyword-based search method, and a radiologist labelled the reports for the presence or absence of nine SIFs. The radiologist’s scores were the reference standard. Each LLM was evaluated with four different prompting strategies: two without Chain-of-Thought (CoT), and two with CoT (one zero shot, one few shot). The automated labelling methods were evaluated against the radiologist primarily using Cohen’s Kappa, and additionally sensitivity, specificity, and F1 scores. Results: GPT-5 achieved substantial or almost perfect agreement for seven out of nine SIFs: aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, abnormality in mediastinal lymph nodes, and thyroid abnormalities, with Cohen κ values of 0.81 (0.70, 0.93), 0.97 (0.93, 1.00), 0.71 (0.62, 0.82), 0.92 (0.87, 0.96), 0.91 (0.84, 0.97), 0.71 (0.62, 0.80), and 0.92 (0.86, 0.98) respectively. For the remaining two SIFs, moderate LLM agreement was obtained with MedGemma-4B-IT and with GPT-4-Turbo with Cohen κ values of 0.48 (0.40, 0.60), 0.42 (0.31, 0.58) for interstitial lung abnormalities and mediastinal mass respectively. The highest agreement was obtained with the non-CoT prompt strategies for most SIFs. Conclusion: This study demonstrated that LLMs can automatically label radiology reports with agreement ranging from almost perfect to moderate, but not one LLM – prompt combination achieved the highest agreement across all SIFs.Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Ministerie van Economische Zaken en Klimaat
This publication has 12 references indexed in Scilit:
- Open-Source Large Language Models in Radiology: A Review and Tutorial for Practical Research and Clinical DeploymentRadiology, 2025
- Privacy-ensuring Open-weights Large Language Models Are Competitive with Closed-weights GPT-4o in Extracting Chest Radiography Findings from Free-Text ReportsRadiology, 2025
- Closing the gap between open source and commercial large language models for medical evidence summarizationnpj Digital Medicine, 2024
- Use of GPT-4 With Single-Shot Learning to Identify Incidental Findings in Radiology ReportsAmerican Journal of Roentgenology, 2024
- Lost in the Middle: How Language Models Use Long ContextsTransactions of the Association for Computational Linguistics, 2024
- Automated image label extraction from radiology reports — A reviewArtificial Intelligence in Medicine, 2024
- ERS/ESTS/ESTRO/ESR/ESTI/EFOMP statement on management of incidental findings from low dose CT screening for lung cancerEuropean Respiratory Journal, 2023
- Feasibility of Using the Privacy-preserving Large Language Model Vicuna for Labeling Radiology ReportsRadiology, 2023
- Significant Incidental Findings in the National Lung Screening TrialJAMA Internal Medicine, 2023
- Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic ScreeningNew England Journal of Medicine, 2011