Large-scale medical image annotation with crowd-powered algorithms
- 8 September 2018
- journal article
- research article
- Published by SPIE-Intl Soc Optical Eng in Journal of Medical Imaging
- Vol. 5 (03) , 034002-14
Abstract
Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.Keywords
Funding Information
- Klaus Tschira Stiftung (SFB/TRR 125)
This publication has 44 references indexed in Scilit:
- The Medical Imaging Interaction Toolkit: challenges and advancesInternational Journal of Computer Assisted Radiology and Surgery, 2013
- Crowd-sourced BioGames: managing the big data problem for next-generation lab-on-a-chip platformsLab on a Chip, 2012
- Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligenceMedical Image Analysis, 2012
- Statistical shape models for 3D medical image segmentation: A reviewMedical Image Analysis, 2009
- LabelMe: A Database and Web-Based Tool for Image AnnotationInternational Journal of Computer Vision, 2007
- The Medical Imaging Interaction ToolkitMedical Image Analysis, 2005
- OsiriX: An Open-Source Software for Navigating in Multidimensional DICOM ImagesJournal of Digital Imaging, 2004
- Liver and Bone Window Settings for Soft-Copy Interpretation of Chest and Abdominal CTAmerican Journal of Roentgenology, 2000
- Individual Comparisons by Ranking MethodsBiometrics Bulletin, 1945
- Measures of the Amount of Ecologic Association Between SpeciesEcology, 1945