Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomas
- 10 January 2024
- journal article
- research article
- Published by BMJ in Journal of Clinical Pathology
- Vol. 78 (3) , 202-207
Abstract
Aims To evaluate the accuracy of Chat Generative Pre-trained Transformer (ChatGPT) powered by GPT-4 in histopathological image detection and classification of colorectal adenomas using the diagnostic consensus provided by pathologists as a reference standard. Methods A study was conducted with 100 colorectal polyp photomicrographs, comprising an equal number of adenomas and non-adenomas, classified by two pathologists. These images were analysed by classic GPT-4 for 1 time in October 2023 and custom GPT-4 for 20 times in December 2023. GPT-4’s responses were compared against the reference standard through statistical measures to evaluate its proficiency in histopathological diagnosis, with the pathologists further assessing the model’s descriptive accuracy. Results GPT-4 demonstrated a median sensitivity of 74% and specificity of 36% for adenoma detection. The median accuracy of polyp classification varied, ranging from 16% for non-specific changes to 36% for tubular adenomas. Its diagnostic consistency, indicated by low kappa values ranging from 0.06 to 0.11, suggested only poor to slight agreement. All of the microscopic descriptions corresponded with their diagnoses. GPT-4 also commented about the limitations in its diagnoses (eg, slide diagnosis best done by pathologists, the inadequacy of single-image diagnostic conclusions, the need for clinical data and a higher magnification view). Conclusions GPT-4 showed high sensitivity but low specificity in detecting adenomas and varied accuracy for polyp classification. However, its diagnostic consistency was low. This artificial intelligence tool acknowledged its diagnostic limitations, emphasising the need for a pathologist’s expertise and additional clinical context.Keywords
This publication has 19 references indexed in Scilit:
- Evaluation of an Artificial Intelligence–Augmented Digital System for Histologic Classification of Colorectal PolypsJAMA Network Open, 2021
- Weak Human Preference Supervision for Deep Reinforcement LearningIEEE Transactions on Neural Networks and Learning Systems, 2021
- Screening for Colorectal CancerJAMA, 2021
- A machine learning-based prognostic predictor for stage III colon cancerScientific Reports, 2020
- Stop explaining black box machine learning models for high stakes decisions and use interpretable models insteadNature Machine Intelligence, 2019
- Digital pathology and artificial intelligenceThe Lancet Oncology, 2019
- Clinicopathological features of colorectal polyps in 2002 and 2012The Korean Journal of Internal Medicine, 2019
- Interobserver variability in assessing dysplasia and architecture in colorectal adenomas: a multicentre Canadian studyJournal of Clinical Pathology, 2014
- Long-Term Colorectal-Cancer Incidence and Mortality after Lower EndoscopyNew England Journal of Medicine, 2013
- Interobserver agreement in the reporting of colorectal polyp pathology among bowel cancer screening pathologists in WalesHistopathology, 2013