Evaluation of the Mayo endoscopic score in ulcerative colitis using a multimodal large language model: a human-blinded accuracy study
Authors:
Open Access
- 31 October 2026
- journal article
- Published by Korean Association for the Study of Intestinal Diseases in Intestinal Research
- Vol. 24 (4) , 724-734
Abstract
Background/Aims: To date, some large language models (LLMs), such as Chat Generative Pre-trained Transformer 4-omni (ChatGPT-4o), can process images through visual transformer patching. An LLM analysis was conducted to assess the ability of ChatGPT-4o to assign the Mayo endoscopic score (MES). Methods: A selection set of high-quality endoscopic frames was identified to compare 4 input models to select the most performant, confirmed in an extended set of 304 frames. Concordance with the evaluation by expert endoscopists was assessed. Results: Only one of the pre-tested models demonstrated significant concordance (κ = 0.232, 95% confidence interval [CI] = 0.167 to 0.296, P= 0.003; ρ = 0.36, P= 0.011), with a mean bias of –0.26 ± 1.192 (95% CI, –2.596 to 2.076). This was confirmed in the extended set (κ = 0.260, 95% CI = 0.195 to 0.324, P< 0.001; ρ = 0.288, P< 0.001). The absolute concordance for the selected model was 44% and 45.3% in the selection and extended sets, respectively. For the identification of moderately-to-severely active disease, a sensitivity of 73% (95% CI, 60% to 82%), specificity of 60% (95% CI, 54% to 66%), positive predictive value of 32% (95% CI, 25% to 40%), and negative predictive value of 90% (95% CI, 84% to 93%) were identified. Conclusions: ChatGPT-4o shows a mild potential in evaluating MES in endoscopic frames, but further refinements are mandatory.Keywords
This publication has 32 references indexed in Scilit:
- Applications of generative artificial intelligence in inflammatory bowel disease: A systematic reviewDigestive and Liver Disease, 2025
- In-context learning enables multimodal large language models to classify cancer pathology imagesNature Communications, 2024
- ‘Fighting fire with fire’ — using LLMs to combat LLM hallucinationsNature, 2024
- Ethical and regulatory challenges of large language models in medicineThe Lancet Digital Health, 2024
- A systematic review and meta-analysis of artificial intelligence-diagnosed endoscopic remission in ulcerative colitisiScience, 2023
- A review on the current status and definitions of activity indices in inflammatory bowel disease: how to use indices for precise evaluationJournal of Gastroenterology, 2022
- Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy imagesScientific Reports, 2022
- World Endoscopy Organization Consensus Statements on Post-Colonoscopy and Post-Imaging Colorectal CancerGastroenterology, 2018
- The Boston bowel preparation scale: a valid and reliable instrument for colonoscopy-oriented researchGastrointestinal Endoscopy, 2009
- Confidence intervals for a ratio of binomial proportions based on paired dataStatistics in Medicine, 2006