Evaluating Large Language Models in Interpreting Cervical Cytology
Preprint
Abstract
Background: Large language models (LLMs) have shown promise in medical imaging, but their utility in cytology remains underexplored. This study evaluates GPT-5 and Gemini 2.5 Pro for Pap smear interpretation. Methods: Digital cervical Pap smear images of 100 cases were obtained from the Hologic Education Site, with Hologic diagnoses considered the gold standard. Representative images were uploaded into GPT-5 and Gemini 2.5 Pro and prompted to provide a diagnosis based on the Third Edition of the Bethesda System for Reporting Cervical Cytopathology. Cases with infectious organisms were assessed using additional images. Concordance was evaluated at exact diagnosis and clinical management groupings, wherein diagnoses with similar management implications were grouped. Sensitivity and specificity for abnormal cytology were also calculated. Results: Concordance of both LLMs for exact diagnostic matches were comparable (GPT-5: 47%, Gemini: 48%) and increased to 66% for clinical management grouping. GPT-5 performed best for low-grade squamous intraepithelial lesions (75%), whereas Gemini 2.5 Pro showed the highest concordance in the high-grade squamous intraepithelial lesion (HSIL) category (82%), although this was largely attributable to its strong tendency to overcall cases as HSIL. Sensitivity for detecting abnormal cytology was 74% for GPT-5 and 84% for Gemini, with specificity of 74% and 71%, respectively. GPT-5 better identified glandular lesions, while Gemini detected organisms more accurately (71% vs. 20%). Conclusions: Current LLMs demonstrate moderate ability to identify cytologic abnormalities but are not yet reliable for independent Pap smear interpretation. Targeted fine-tuning, prompt optimization, and cytology-specific training could enhance their utility as adjunctive tools in cytology workflows.Keywords
This publication has 13 references indexed in Scilit:
- Evaluation of general-purpose large language models as diagnostic support tools in cervical cytologyPathology - Research and Practice, 2025
- A systematic review of large language model (LLM) evaluations in clinical medicineBMC Medical Informatics and Decision Making, 2025
- Encouragement vs. liability: How prompt engineering influences ChatGPT-4's radiology exam performanceClinical Imaging, 2024
- Thinking like a pathologist: Morphologic approach to hepatobiliary tumors by ChatGPTAmerican Journal of Clinical Pathology, 2024
- Validation of AI-assisted ThinPrep® Pap test screening using the GeniusTM Digital Diagnostics SystemJournal of Pathology Informatics, 2024
- Improving the use of LLMs in radiology through prompt engineering: from precision prompts to zero-shot learningRöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren, 2024
- ChatGPT’s innovative application in blood morphology recognitionJournal of the Chinese Medical Association, 2024
- Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomasJournal of Clinical Pathology, 2024
- Evaluation of ChatGPT pathology knowledge using board-style questionsAmerican Journal of Clinical Pathology, 2023
- Interpretation pitfalls and malignant mimics in cervical cytologyJournal of the American Society of Cytopathology, 2021