Christopher J. Brady

Purpose: To assess the accuracy of crowdsourcing for grading optic nerve images for glaucoma using Amazon Mechanical Turk before and after training modules. Materials and Methods: Images (n=60) from 2 large population studies were graded for glaucoma status and vertical cup-to-disc ratio (VCDR). In the baseline trial, users on Amazon Mechanical Turk (Turkers) graded fundus photos for glaucoma and VCDR after reviewing annotated example images. In 2 additional trials, Turkers viewed a 26-slide PowerPoint training or a 10-minute video training and passed a quiz before being permitted to grade the same 60 images. Each image was graded by 10 unique Turkers in all trials. The mode of Turker grades for each image was compared with an adjudicated expert grade to determine accuracy as well as the sensitivity and specificity of Turker grading. Results: In the baseline study, 50% of the images were graded correctly for glaucoma status and the area under the receiver operating characteristic (AUROC) was 0.75 [95% confidence interval (CI), 0.64-0.87]. Post-PowerPoint training, 66.7% of the images were graded correctly with AUROC of 0.86 (95% CI, 0.78-0.95). Finally, Turker grading accuracy was 63.3% with AUROC of 0.89 (95% CI, 0.83-0.96) after video training. Overall, Turker VCDR grades for each image correlated with expert VCDR grades (Bland-Altman plot mean difference=−0.02). Conclusions: Turkers graded 60 fundus images quickly and at low cost, with grading accuracy, sensitivity, and specificity, all improving with brief training. With effective education, crowdsourcing may be an efficient tool to aid in the identification of glaucomatous changes in retinal images.

It has now been well established that a nonmydriatic camera located well outside the ophthalmologist or optometrist office can be used to capture images of the retinal fundus for remote interpretation. In the setting of diabetic retinopathy, this “store-and-forward” paradigm has been validated against clinical examination and as well as the research gold standard 7-field Early Treatment Diabetic Retinopathy Study images¹; it has been demonstrated that remotely acquired images can allow clinicians to reach the same diagnosis and management plan as these more traditional methods. While there have been questions about the possible role for ultrawidefield imaging¹ and optical coherence tomography in ocular telehealth paradigms,² monoscopic photographic imaging has become the foundation of large diabetic retinopathy screening programs.³

Purpose of review Crowdsourcing involves the use of the collective intelligence of online communities to produce solutions and outcomes for defined objectives. The use of crowdsourcing is growing in many scientific areas. Crowdsourcing in ophthalmology has been used in basic science and clinical research; however, it also shows promise as a method with wide-ranging applications. This review presents current findings on the use of crowdsourcing in ophthalmology and potential applications in the future. Recent findings Crowdsourcing has been used to distinguish normal retinal images from images with diabetic retinopathy; the collective intelligence of the crowd was able to correctly classify 81% of 230 images (19 unique) for US$1.10/eye in 20 min. Crowdsourcing has also been used to distinguish normal optic discs from abnormal ones with reasonable sensitivity (83–88%), but low specificity (35–43%). Another study used crowdsourcing for quick and reliable manual segmentation of optical coherence tomography images. Outside of ophthalmology, crowdsourcing has been used for text and image interpretation, language translation, and data analysis. Summary Crowdsourcing has the potential for rapid and economical data processing. Among other applications, it could be used in research settings to provide the ‘ground-truth’ data, and in the clinical settings to relieve the burden of image processing on experts.

This commentary discusses a study published in JAMA Ophthalmology that describes the accuracy of a smartphone-based visual acuity test for detecting visual impa

To the Editor We read with interest Rahimy and Garg’s experience using Google Glass to film a scleral buckling procedure. We agree there is a need for better vi