A three-dimensional multi-modal foundation model for optical coherence tomography
- 24 April 2026
- journal article
- Published by Springer Nature in Nature Biomedical Engineering
Abstract
Vision loss caused by retinal diseases remains a leading global cause of disability. Optical coherence tomography (OCT) is an imaging technique that is used for diagnosing retinal diseases. Computational models can use OCT images for various diagnostic and prognostic tasks, but most existing approaches fail to fully leverage the rich three-dimensional (3D) structure of OCT data and lack the capability to integrate other retinal imaging modalities into the analysis. Here, to address these limitations, we present OCTCube-M, a 3D OCT-based multi-modal framework designed for the integrated analysis of 3D OCT and 2D en face (EF) images. OCTCube-M exploits COEP, an effective multi-modal contrastive learning method, to integrate OCT with other retinal imaging modalities, such as fundus autofluorescence imaging and infrared retinal imaging (IR). Using the OCTCube-M framework, we developed three models: OCTCube (uni-modal), OCTCube-IR (bi-modal) and OCTCube-EF (tri-modal). OCTCube, a 3D foundation model pre-trained on 26,605 3D OCT volumes comprising 1.62 million 2D OCT slices, achieved state-of-the-art performance in predicting 8 retinal diseases while demonstrating robust generalizability across cohorts, devices and modalities. OCTCube-IR extends OCTCube by incorporating 26,685 pairs of OCT and IR images, enabling accurate cross-modality retrieval and joint analysis of these two modalities. OCTCube-EF, trained on over 4 million 2D OCT slices and 400 thousand EF retinal images, excels in predicting the growth rate of geographic atrophy across datasets collected from 6 multi-centre clinical trials across 23 countries. Collectively, OCTCube-M is a 3D multi-modal foundation model framework for integrating OCT and other retinal imaging modalities. It demonstrated substantial advancements in cross-site, cross-device, cross-modality and systemic disease prediction, while offering substantial utility in geographic atrophy clinical trials.Keywords
This publication has 73 references indexed in Scilit:
- Economic Burden of Late-Stage Age-Related Macular Degeneration in Bulgaria, Germany, and the USJAMA Ophthalmology, 2024
- UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling2024
- Changes in wider field swept-source OCT angiography vascular metrics with anti-vascular endothelial growth factor therapy in central retinal vein occlusionGraefe's Archive for Clinical and Experimental Ophthalmology, 2024
- Generative adversarial network-based deep learning approach in classification of retinal conditions with optical coherence tomography imagesGraefe's Archive for Clinical and Experimental Ophthalmology, 2022
- Automated diagnosis of age‐related macular degeneration using multi‐modal vertical plane feature fusion via deep learningMedical Physics, 2022
- A Global Assessment of Eye Health and Quality of LifeJAMA Ophthalmology, 2021
- OCT Imaging in Glaucoma2021
- Comparison of the Iowa Reference Algorithm to the Heidelberg Spectralis optical coherence tomography segmentation algorithmJournal of Biophotonics, 2020
- Efficacy and Safety of Lampalizumab for Geographic Atrophy Due to Age-Related Macular DegenerationJAMA Ophthalmology, 2018
- Optical Coherence Tomography2012