Can off-the-shelf visual large language models detect and diagnose ocular diseases from retinal photographs?
Authors:
- Sahana SrinivasanSSSahana Srinivasan
- Hongwei JiHJHongwei Ji
- David Ziyou ChenDCDavid Ziyou Chen
- Wendy WongWWWendy Wong
- Zhi Da SohZSZhi Da Soh
- Jocelyn Hui Lin GohJGJocelyn Hui Lin Goh
- Krithi PushpanathanKPKrithi Pushpanathan
- Xiaofei WangXiaofei Wang
- Weizhi MaWMWeizhi Ma
- Tien Yin WongTWTien Yin Wong
- Ya Xing WangYWYa Xing Wang
- Ching-Yu ChengCCChing-Yu Cheng
Open Access
- 7 April 2025
- journal article
- retina
- Published by BMJ in BMJ Open Ophthalmology
- Vol. 10 (1) , e002076
Abstract
Background: The advent of generative artificial intelligence has led to the emergence of multiple vision large language models (VLLMs). This study aimed to evaluate the capabilities of commonly available VLLMs, such as OpenAI’s GPT-4V and Google’s Gemini, in detecting and diagnosing ocular diseases from retinal images. Methods and analysis: From the Singapore Epidemiology of Eye Diseases (SEED) study, we selected 44 representative retinal photographs, including 10 healthy and 34 representing six eye diseases (age-related macular degeneration, diabetic retinopathy, glaucoma, visually significant cataract, myopic macular degeneration and retinal vein occlusion). OpenAI’s GPT-4V (both default and data analyst modes) and Google Gemini were prompted with each image to determine if the retina was normal or abnormal and to provide diagnostic descriptions if deemed abnormal. The outputs from the VLLMs were evaluated for accuracy by three attending-level ophthalmologists using a three-point scale (poor, borderline, good). Results: GPT-4V default mode demonstrated the highest detection rate, correctly identifying 33 out of 34 detected correctly (97.1%), outperforming its data analyst mode (61.8%) and Google Gemini (41.2%). Despite the relatively high detection rates, the quality of diagnostic descriptions was generally suboptimal—with only 21.2% of GPT-4V’s (default) responses, 4.8% of GPT-4V’s (data analyst) responses and 28.6% for Google Gemini’s responses rated as good. Conclusions: Although GPT-4V default mode showed generally high sensitivity in abnormality detection, all evaluated VLLMs were inadequate in providing accurate diagnoses for ocular diseases. These findings emphasise the need for domain-customised VLLMs and suggest the continued need for human oversight in clinical ophthalmology.Keywords
Funding Information
- National Tsing Hua University (BX20230189)
- National Medical Research Council (NMRC/MOH/ HCSAINV21nov-000)
This publication has 15 references indexed in Scilit:
- Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for OphthalmologyJAMA Ophthalmology, 2024
- Analysis of ChatGPT Responses to Ophthalmic Cases: Can ChatGPT Think like an Ophthalmologist?Ophthalmology Science, 2024
- Vision-Language Models for Feature Detection of Macular Diseases on Optical Coherence TomographyJAMA Ophthalmology, 2024
- Large language models and their impact in ophthalmologyThe Lancet Digital Health, 2023
- Clinician-Driven AI: Code-Free Self-Training on Public Data for Diabetic Retinopathy ReferralJAMA Ophthalmology, 2023
- Artificial intelligence and digital health in global eye health: opportunities and challengesThe Lancet Global Health, 2023
- Large language models in medicineNature Medicine, 2023
- Evaluating the Performance of ChatGPT in OphthalmologyOphthalmology Science, 2023
- ChatGPT and Ophthalmology: Exploring Its Potential with Discharge Summaries and Operative NotesSeminars in Ophthalmology, 2023
- Cohort Profile: The Singapore Epidemiology of Eye Diseases study (SEED)International Journal of Epidemiology, 2021