Evaluation of an AI system for the detection of diabetic retinopathy from images captured with a handheld portable fundus camera: the MAILOR AI study
- 7 May 2020
- journal article
- research article
- Published by Springer Nature in Eye
- Vol. 35 (2) , 632-638
Abstract
Objectives To evaluate the performance of an artificial intelligence (AI) system (Pegasus, Visulytix Ltd., UK*) at the detection of diabetic retinopathy (DR) from images captured by a handheld portable fundus camera. Methods A cohort of 6404 patients (~80% with diabetes mellitus) was screened for retinal diseases using a handheld portable fundus camera (Pictor Plus, Volk Optical Inc., USA) at the Mexican Advanced Imaging Laboratory for Ocular Research. The images were graded for DR by specialists according to the Scottish DR grading scheme. The performance of the AI system was evaluated, retrospectively, in assessing referable DR (RDR) and proliferative DR (PDR) and compared with the performance on a publicly available desktop camera benchmark dataset. Results For RDR detection, Pegasus performed with an 89.4% (95% CI: 88.0–90.7) area under the receiver operating characteristic (AUROC) curve for the MAILOR cohort, compared with an AUROC of 98.5% (95% CI: 97.8–99.2) on the benchmark dataset. This difference was statistically significant. Moreover, no statistically significant difference was found in performance for PDR detection with Pegasus achieving an AUROC of 94.3% (95% CI: 91.0–96.9) on the MAILOR cohort and 92.2% (95% CI: 89.4–94.8) on the benchmark dataset. Conclusions Pegasus showed good transferability for the detection of PDR from a curated desktop fundus camera dataset to real-world clinical practice with a handheld portable fundus camera. However, there was a substantial, and statistically significant, decrease in the diagnostic performance for RDR when using the handheld device.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening ResearchData, 2018
- Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligenceEye, 2018
- Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With DiabetesJAMA, 2017
- Quality and learning curve of handheld versus stand-alone non-mydriatic camerasClinical Ophthalmology, 2017
- Deep Learning in MammographyInvestigative Radiology, 2017
- Automated Diabetic Retinopathy Image Assessment SoftwareOphthalmology, 2017
- Dermatologist-level classification of skin cancer with deep neural networksNature, 2017
- Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus PhotographsJAMA, 2016
- The number of ophthalmologists in practice and training worldwide: a growing gap despite more than 200 000 practitionersBritish Journal of Ophthalmology, 2012
- Retinal Imaging and Image AnalysisIEEE Reviews in Biomedical Engineering, 2010