Benchmark Evaluation of Vision-Enabled Large Language Models in Mammography: A Case–Control Study
- 21 October 2026
- journal article
- Published by Oxford University Press (OUP) in Journal of Breast Imaging
Abstract
Objective: To compare the diagnostic performance of five contemporary vision-enabled large language models (LLMs) in mammography interpretation using a case–control benchmark design. Methods: A total of 200 cases were analyzed using a quota-balanced case–control design (100 malignant, BI-RADS 4–5; 100 controls, BI-RADS 1–2). All models (ChatGPT-5T, ChatGPT-5I, ChatGPT-4o, Gemini 2.5 Pro, and Gemini 2.5 Flash) received standard mammography images. Models were evaluated for lesion detection, laterality, quadrant localization, lesion type, and malignant–benign discrimination. Performance was assessed using accuracy, sensitivity, specificity, balanced accuracy, Cohen’s κ, and area under the receiver operating characteristic curve (AUC). Breast density was recorded for all cases, while histopathological subtype and lesion size were documented for malignant cases. Results: Lesion detection accuracy ranged from 79% to 91%, with high malignant sensitivity (92%–100%) but lower specificity in normal cases (24%–82%). ChatGPT-5T performed best in laterality (83%; κ = 0.70) and quadrant localization (73%; κ = 0.62). Lesion type classification was modest (74%–85%). In malignant–benign discrimination, ChatGPT-5T achieved the highest balanced accuracy (87%), while ChatGPT-4o showed poor specificity (28%). The study cohort was predominantly composed of lower-density breasts, larger lesions, and invasive ductal carcinoma as the most common subtype. Conclusion: Vision-enabled LLMs demonstrated high sensitivity for breast lesion detection and classification tasks. More deliberative models, particularly ChatGPT-5T, consistently outperformed faster counterparts, especially in spatial reasoning tasks. However, reduced specificity in normal cases underscores that current LLM outputs should remain under radiologist supervision as supportive second-reader tools rather than standalone diagnostic systems. Given the quota-balanced design and cohort composition, results should be interpreted as comparative benchmarking rather than real-world diagnostic performance.Keywords
This publication has 16 references indexed in Scilit:
- Performance Comparison of Cutting-Edge Large Language Models on the ACR In-Training Examination: An Update for 2025Academic Radiology, 2025
- One Year On: Assessing Progress of Multimodal Large Language Model Performance on RSNA 2024 Case of the Day QuestionsRadiology, 2025
- A preliminary investigation into the potential, pitfalls, and limitations of large language models for mammography interpretationDiscover Oncology, 2025
- Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research ApplicationsRadiology, 2024
- Automatic classification and prioritisation of actionable BI-RADS categories using natural language processing modelsClinical Radiology, 2023
- Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging PilotJournal of the American College of Radiology, 2023
- Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and LimitationsRadiology, 2023
- Staff shortages are putting UK breast cancer screening “at risk,” survey findsBMJ, 2016
- Effect of mammographic screening from age 40 years on breast cancer mortality in the UK Age trial at 17 years' follow-up: a randomised controlled trialThe Lancet Oncology, 2015
- Breast-Cancer Screening — Viewpoint of the IARC Working GroupNew England Journal of Medicine, 2015