Xuefei Song
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Xuefei Song has a total of 187 co-authors. Here are the top co-authors by number of shared publications:
- Huifang Zhou
Shanghai Jiao Tong University School of Medicine
20 shared publications - Xianqun Fan
Shanghai Jiao Tong University School of Medicine
18 shared publications Haiyang Zhang
Shanghai Jiao Tong University School of Medicine
17 shared publicationsMengda Jiang
Shanghai Jiao Tong University School of Medicine
10 shared publicationsYinwei Li
State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, People’s Republic of China
9 shared publicationsJing Sun
State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, People’s Republic of China
8 shared publications
Latest Publications
- Open Access
- 26 August 2026
- Published by Springer Nature in Scientific Data
A volumetric orbital computed tomography dataset with expert-annotated radiological findings
- 26 August 2026
- Published by Springer Nature in Scientific Data
Open AccessA volumetric orbital computed tomography dataset with expert-annotated radiological findings
- Open Access
- 1 February 2026
- Published by SAGE Publications in Digital Health
- Vol. 12
RDINet: A novel dynamic mapping model integrating radiomics and deep learning for predicting treatment response in thyroid eye disease
Objective: Artificial intelligence-based magnetic resonance imaging (MRI) analysis has shown promising potential in predicting the response to intravenous glucocorticoid (IVGC) treatment in thyroid eye disease (TED). We aimed to explore a novel model for multimodal feature fusion based on MRI data to enhance predictive effectiveness. Methods: We enrolled a total of 147 TED patients who underwent IVGC treatment. Pre-treatment orbital T2-weighted image was obtained for each subject, and individual segmentation of four extraocular muscles (EOMs) was performed. Radiomics analysis using different combinations of muscles as region of interests (ROIs) and machine learning algorithms were employed. Furthermore, we proposed a Residual Dynamic Integration Network (RDINet), which integrated image features extracted by convolutional neural network (ResNet-50), radiomics features, and clinical features using a dynamic mapping module. Results: Combining medial rectus (MR) and lateral rectus (LR) in the modeling exhibited marginally improved performance than using all four EOMs in radiomics analysis (AUC = 0.8981 vs. 0.8796). ResNet-50 models based on MR + LR and four EOMs yielded AUC values of 0.9074 and 0.8889, respectively. RDINet achieved AUC values of 0.9213 and 0.9028 for the above two ROI strategies, which increased to 0.9537 and 0.9259 after integrating clinical features into modeling. Conclusion: The focused combination of MR and LR in radiomics and deep learning analysis outperformed the inclusion of four EOMs for IVGC response prediction in TED patients. The proposed RDINet realizes effective multimodal feature fusion of TED, which serves as a potent dynamic mapping model for treatment response prediction.
- 1 February 2026
- Published by SAGE Publications in Digital Health
- Vol. 12
Open AccessRDINet: A novel dynamic mapping model integrating radiomics and deep learning for predicting treatment response in thyroid eye disease
Objective: Artificial intelligence-based magnetic resonance imaging (MRI) analysis has shown promising potential in predicting the response to intravenous glucocorticoid (IVGC) treatment in thyroid eye disease (TED). We aimed to explore a novel model for multimodal feature fusion based on MRI data to enhance predictive effectiveness. Methods: We enrolled a total of 147 TED patients who underwent IVGC treatment. Pre-treatment orbital T2-weighted image was obtained for each subject, and individual segmentation of four extraocular muscles (EOMs) was performed. Radiomics analysis using different combinations of muscles as region of interests (ROIs) and machine learning algorithms were employed. Furthermore, we proposed a Residual Dynamic Integration Network (RDINet), which integrated image features extracted by convolutional neural network (ResNet-50), radiomics features, and clinical features using a dynamic mapping module. Results: Combining medial rectus (MR) and lateral rectus (LR) in the modeling exhibited marginally improved performance than using all four EOMs in radiomics analysis (AUC = 0.8981 vs. 0.8796). ResNet-50 models based on MR + LR and four EOMs yielded AUC values of 0.9074 and 0.8889, respectively. RDINet achieved AUC values of 0.9213 and 0.9028 for the above two ROI strategies, which increased to 0.9537 and 0.9259 after integrating clinical features into modeling. Conclusion: The focused combination of MR and LR in radiomics and deep learning analysis outperformed the inclusion of four EOMs for IVGC response prediction in TED patients. The proposed RDINet realizes effective multimodal feature fusion of TED, which serves as a potent dynamic mapping model for treatment response prediction.
- Open Access
- 24 December 2025
- Published by MDPI AG in Bioengineering
- Vol. 13 (1) , 12
DME-RWKV: An Interpretable Multimodal Deep Learning Framework for Predicting Anti-VEGF Response in Diabetic Macular Edema
Diabetic macular edema (DME) is a leading cause of vision loss, and predicting patients’ response to anti-vascular endothelial growth factor (anti-VEGF) therapy remains a clinical challenge. In this study, we developed an interpretable deep learning model for treatment prediction and biomarker analysis. We retrospectively analyzed 402 eyes from 371 patients with DME. The proposed DME-Receptance Weighted Key Value (RWKV) integrates optical coherence tomography (OCT) and ultra-widefield (UWF) imaging using Causal Attention Learning (CAL), curriculum learning, and global completion (GC) loss to enhance microlesion detection and structural consistency. The model achieved a Dice coefficient of 71.91 ± 8.50% for OCT biomarker segmentation and an AUC of 84.36% for predicting anti-VEGF response, outperforming state-of-the-art methods. By mimicking clinical reasoning with multimodal integration, DME-RWKV demonstrated strong interpretability and robustness, providing a promising AI framework for precise and explainable prediction of anti-VEGF treatment outcomes in DME.
- 24 December 2025
- Published by MDPI AG in Bioengineering
- Vol. 13 (1) , 12
Open AccessDME-RWKV: An Interpretable Multimodal Deep Learning Framework for Predicting Anti-VEGF Response in Diabetic Macular Edema
Diabetic macular edema (DME) is a leading cause of vision loss, and predicting patients’ response to anti-vascular endothelial growth factor (anti-VEGF) therapy remains a clinical challenge. In this study, we developed an interpretable deep learning model for treatment prediction and biomarker analysis. We retrospectively analyzed 402 eyes from 371 patients with DME. The proposed DME-Receptance Weighted Key Value (RWKV) integrates optical coherence tomography (OCT) and ultra-widefield (UWF) imaging using Causal Attention Learning (CAL), curriculum learning, and global completion (GC) loss to enhance microlesion detection and structural consistency. The model achieved a Dice coefficient of 71.91 ± 8.50% for OCT biomarker segmentation and an AUC of 84.36% for predicting anti-VEGF response, outperforming state-of-the-art methods. By mimicking clinical reasoning with multimodal integration, DME-RWKV demonstrated strong interpretability and robustness, providing a promising AI framework for precise and explainable prediction of anti-VEGF treatment outcomes in DME.
- Open Access
- 20 November 2025
- Published by Springer Nature in npj Digital Medicine
- Vol. 8 (1) , 705
ROFI: a deep learning-based ophthalmic sign-preserving and reversible patient face anonymizer
Patient face images provide a convenient mean for evaluating eye diseases, while also raising privacy concerns. Here, we introduce ROFI, a deep learning-based privacy protection framework for ophthalmology. Using weakly supervised learning and neural identity translation, ROFI anonymizes facial features while retaining disease features (over 98% accuracy, κ > 0.90). It achieves 100% diagnostic sensitivity and high agreement (κ > 0.90) across eleven eye diseases in three cohorts, anonymizing over 95% of images. ROFI works with AI systems, maintaining original diagnoses (κ > 0.80), and supports secure image reversal (over 98% similarity), enabling audits and long-term care. These results show ROFI's effectiveness of protecting patient privacy in the digital medicine era.
- 20 November 2025
- Published by Springer Nature in npj Digital Medicine
- Vol. 8 (1) , 705
Open AccessROFI: a deep learning-based ophthalmic sign-preserving and reversible patient face anonymizer
Patient face images provide a convenient mean for evaluating eye diseases, while also raising privacy concerns. Here, we introduce ROFI, a deep learning-based privacy protection framework for ophthalmology. Using weakly supervised learning and neural identity translation, ROFI anonymizes facial features while retaining disease features (over 98% accuracy, κ > 0.90). It achieves 100% diagnostic sensitivity and high agreement (κ > 0.90) across eleven eye diseases in three cohorts, anonymizing over 95% of images. ROFI works with AI systems, maintaining original diagnoses (κ > 0.80), and supports secure image reversal (over 98% similarity), enabling audits and long-term care. These results show ROFI's effectiveness of protecting patient privacy in the digital medicine era.
- Top Cited PapersOpen Access
- 27 October 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e76947
Evaluating Large Language Models in Ophthalmology: Systematic Review
Background: Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. Objective: This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. Methods: A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. Results: Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). Conclusions: Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
- 27 October 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e76947
Top Cited PapersOpen AccessEvaluating Large Language Models in Ophthalmology: Systematic Review
Background: Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. Objective: This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. Methods: A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. Results: Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). Conclusions: Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
- 20 October 2025
- Published by BMJ in British Journal of Ophthalmology
- Vol. 110 (6) , 661-668
Enhancing diabetic retinopathy diagnosis and grading: a retrospective study on AI-assisted decision making and cost analysis
Background/Aims Diabetic retinopathy (DR) is a major ocular complication of diabetes mellitus. While artificial intelligence (AI)-based DR screening tools have gained widespread adoption, most research focuses on comparing AI performance with human, with limited attention to AI’s role as assistants. This study evaluates the impact of AI-assisted decision-making on DR diagnosis and grading based on colour fundus photographs (CFP) and ultra-widefield fundus (UWF) images. Methods A total of 224 retinal images were analysed by 21 ophthalmologists and primary care physicians (PCPs) in China. Participants independently diagnosed and graded DR based on CFP and UWF images. After a 1-week interval, they repeated the task with AI assistance. Diagnosis accuracy was compared with a gold standard before and after AI assistance. Incremental costs and accuracy improvements were assessed using generalized estimating equations (GEE) models. Results AI assistance significantly improved DR diagnosis accuracy for both CFP and UWF images. For CFP, accuracy increased from 79.90% to 85.68% for PCPs, 81.19% to 88.69% for ophthalmic residents and 81.41% to 88.05% for ophthalmic attendings. Similar improvements were observed for UWF, with accuracy rising from 83.62% to 89.66% for residents and from 81.31% to 88.98% for attendings. GEE analysis revealed an incremental cost of 4.79 units and an accuracy improvement of 0.35 units with AI assistance. Conclusion AI assistance shows potential in improving the accuracy of DR diagnosis and grading. Despite the associated costs, AI enables ophthalmologists to achieve superior diagnosis, facilitating earlier DR detection and treatment.
- 20 October 2025
- Published by BMJ in British Journal of Ophthalmology
- Vol. 110 (6) , 661-668
Enhancing diabetic retinopathy diagnosis and grading: a retrospective study on AI-assisted decision making and cost analysis
Background/Aims Diabetic retinopathy (DR) is a major ocular complication of diabetes mellitus. While artificial intelligence (AI)-based DR screening tools have gained widespread adoption, most research focuses on comparing AI performance with human, with limited attention to AI’s role as assistants. This study evaluates the impact of AI-assisted decision-making on DR diagnosis and grading based on colour fundus photographs (CFP) and ultra-widefield fundus (UWF) images. Methods A total of 224 retinal images were analysed by 21 ophthalmologists and primary care physicians (PCPs) in China. Participants independently diagnosed and graded DR based on CFP and UWF images. After a 1-week interval, they repeated the task with AI assistance. Diagnosis accuracy was compared with a gold standard before and after AI assistance. Incremental costs and accuracy improvements were assessed using generalized estimating equations (GEE) models. Results AI assistance significantly improved DR diagnosis accuracy for both CFP and UWF images. For CFP, accuracy increased from 79.90% to 85.68% for PCPs, 81.19% to 88.69% for ophthalmic residents and 81.41% to 88.05% for ophthalmic attendings. Similar improvements were observed for UWF, with accuracy rising from 83.62% to 89.66% for residents and from 81.31% to 88.98% for attendings. GEE analysis revealed an incremental cost of 4.79 units and an accuracy improvement of 0.35 units with AI assistance. Conclusion AI assistance shows potential in improving the accuracy of DR diagnosis and grading. Despite the associated costs, AI enables ophthalmologists to achieve superior diagnosis, facilitating earlier DR detection and treatment.
- Open Access
ROFI: A Deep Learning-Based Ophthalmic Sign-Preserving and Reversible Patient Face Anonymizer
Patient face images provide a convenient mean for evaluating eye diseases, while also raising privacy concerns. Here, we introduce ROFI, a deep learning-based privacy protection framework for ophthalmology. Using weakly supervised learning and neural identity translation, ROFI anonymizes facial features while retaining disease features (over 98\% accuracy, $κ> 0.90$). It achieves 100\% diagnostic sensitivity and high agreement ($κ> 0.90$) across eleven eye diseases in three cohorts, anonymizing over 95\% of images. ROFI works with AI systems, maintaining original diagnoses ($κ> 0.80$), and supports secure image reversal (over 98\% similarity), enabling audits and long-term care. These results show ROFI's effectiveness of protecting patient privacy in the digital medicine era.
Open AccessROFI: A Deep Learning-Based Ophthalmic Sign-Preserving and Reversible Patient Face Anonymizer
Patient face images provide a convenient mean for evaluating eye diseases, while also raising privacy concerns. Here, we introduce ROFI, a deep learning-based privacy protection framework for ophthalmology. Using weakly supervised learning and neural identity translation, ROFI anonymizes facial features while retaining disease features (over 98\% accuracy, $κ> 0.90$). It achieves 100\% diagnostic sensitivity and high agreement ($κ> 0.90$) across eleven eye diseases in three cohorts, anonymizing over 95\% of images. ROFI works with AI systems, maintaining original diagnoses ($κ> 0.80$), and supports secure image reversal (over 98\% similarity), enabling audits and long-term care. These results show ROFI's effectiveness of protecting patient privacy in the digital medicine era.
- Open Access
- 1 August 2025
- Published by Taylor & Francis in Journal of Inflammation Research
- Vol. ume 18, 10489-10498
Glymphatic System Dysfunction in Thyroid Eye Disease Associated with Disease Activity and Duration
Purpose: To investigate the alterations in the glymphatic system in patients with thyroid eye disease (TED) using diffusion tensor imaging (DTI) analysis along the perivascular space (ALPS) and to explore the correlation between the glymphatic system and clinical features of TED. Methods: The study included 47 patients with TED, including 20 active TED patients (AP) and 27 inactive TED patients (IP), along with 24 healthy controls (HC). Imaging data including DTI sequence were acquired using a 3.0 Tesla scanner. ALPS values were calculated based on the diffusivity of the bilateral superior corona radiata and superior longitudinal fasciculus. Correlation analysis were thus performed between ALPS values and clinical characteristics. Results: The DTI-ALPS analysis in the TED patient group collectively showed lower mean and left ALPS values compared to the HC, with statistically significant differences observed (mean ALPS: P=0.0308, left ALPS: P=0.0032). Among the TED subgroups, the IP had significantly lower left ALPS values than HC (P=0.006). Correlation analysis within the IP subgroup indicated that lower ALPS values were significantly associated with longer disease duration (P=0.015) and greater palpebral fissure height (P=0.028). Conclusions: Glymphatic system dysfunction is evident in TED patients, with its extent influenced by disease activity and duration.
- 1 August 2025
- Published by Taylor & Francis in Journal of Inflammation Research
- Vol. ume 18, 10489-10498
Open AccessGlymphatic System Dysfunction in Thyroid Eye Disease Associated with Disease Activity and Duration
Purpose: To investigate the alterations in the glymphatic system in patients with thyroid eye disease (TED) using diffusion tensor imaging (DTI) analysis along the perivascular space (ALPS) and to explore the correlation between the glymphatic system and clinical features of TED. Methods: The study included 47 patients with TED, including 20 active TED patients (AP) and 27 inactive TED patients (IP), along with 24 healthy controls (HC). Imaging data including DTI sequence were acquired using a 3.0 Tesla scanner. ALPS values were calculated based on the diffusivity of the bilateral superior corona radiata and superior longitudinal fasciculus. Correlation analysis were thus performed between ALPS values and clinical characteristics. Results: The DTI-ALPS analysis in the TED patient group collectively showed lower mean and left ALPS values compared to the HC, with statistically significant differences observed (mean ALPS: P=0.0308, left ALPS: P=0.0032). Among the TED subgroups, the IP had significantly lower left ALPS values than HC (P=0.006). Correlation analysis within the IP subgroup indicated that lower ALPS values were significantly associated with longer disease duration (P=0.015) and greater palpebral fissure height (P=0.028). Conclusions: Glymphatic system dysfunction is evident in TED patients, with its extent influenced by disease activity and duration.
- 24 July 2025
- Published by Springer Nature in Endocrine
- Vol. 90 (2) , 719-729
Effectiveness of intravenous glucocorticoids in active thyroid eye disease patients with low clinical activity scores
Purpose To assess and predict the therapeutic efficacy of intravenous glucocorticoids (IVGC) in thyroid eye disease (TED) patients with clinical activity score (CAS) < 3, but presenting magnetic resonance imaging (MRI)-confirmed intraorbital inflammation. Methods We retrospectively analyzed the clinical data of 91 active TED patients with low CAS that received IVGC treatment (4.5 g, 12 weeks). In terms of treatment response evaluation, the EUGOGO standard (Two-Item Standard) and a One-Item Standard modified on this basis were implemented. Univariate and multivariable logistic regression analyses were used to establish prediction models. Receiver operating characteristic (ROC) curve analysis was performed and the area under the curve (AUC) was calculated. Results Under Two-Item Standard, 31 of the 91 patients (34.1%) were determined as responsive to IVGC, and 60 (65.9%) were unresponsive. MRD-1 was significantly different between responsive and unresponsive groups (P-value < 0.05). Under One-Item Standard, 43 (47.3%) were responsive, and 48 (52.7%) were unresponsive. MRD-2 and exophthalmos were significantly different between two groups (P-value < 0.05). By implementing multivariable regression, the reliability of predicting treatment response of IVGC in active TED patients with low CAS reached AUC = 0.709 under the Two-Item Standard and AUC = 0.792 under the One-Item Standard. Conclusion It is suggested that MRI-based intraorbital active TED patients with low CAS and significant symptoms should be considered for anti-inflammatory therapy, particularly those with specific clinical and radiological features. Clinical assessment and radiological evaluations are valuable for predicting IVGC effectiveness and achieving precision treatment.
- 24 July 2025
- Published by Springer Nature in Endocrine
- Vol. 90 (2) , 719-729
Effectiveness of intravenous glucocorticoids in active thyroid eye disease patients with low clinical activity scores
Purpose To assess and predict the therapeutic efficacy of intravenous glucocorticoids (IVGC) in thyroid eye disease (TED) patients with clinical activity score (CAS) < 3, but presenting magnetic resonance imaging (MRI)-confirmed intraorbital inflammation. Methods We retrospectively analyzed the clinical data of 91 active TED patients with low CAS that received IVGC treatment (4.5 g, 12 weeks). In terms of treatment response evaluation, the EUGOGO standard (Two-Item Standard) and a One-Item Standard modified on this basis were implemented. Univariate and multivariable logistic regression analyses were used to establish prediction models. Receiver operating characteristic (ROC) curve analysis was performed and the area under the curve (AUC) was calculated. Results Under Two-Item Standard, 31 of the 91 patients (34.1%) were determined as responsive to IVGC, and 60 (65.9%) were unresponsive. MRD-1 was significantly different between responsive and unresponsive groups (P-value < 0.05). Under One-Item Standard, 43 (47.3%) were responsive, and 48 (52.7%) were unresponsive. MRD-2 and exophthalmos were significantly different between two groups (P-value < 0.05). By implementing multivariable regression, the reliability of predicting treatment response of IVGC in active TED patients with low CAS reached AUC = 0.709 under the Two-Item Standard and AUC = 0.792 under the One-Item Standard. Conclusion It is suggested that MRI-based intraorbital active TED patients with low CAS and significant symptoms should be considered for anti-inflammatory therapy, particularly those with specific clinical and radiological features. Clinical assessment and radiological evaluations are valuable for predicting IVGC effectiveness and achieving precision treatment.
- Open Access
- 5 May 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Evaluating Large Language Models in Ophthalmology: Systematic Review (Preprint)
BACKGROUND Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. OBJECTIVE This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. METHODS A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. RESULTS Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). CONCLUSIONS Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
- 5 May 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessEvaluating Large Language Models in Ophthalmology: Systematic Review (Preprint)
BACKGROUND Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. OBJECTIVE This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. METHODS A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. RESULTS Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). CONCLUSIONS Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.