Robustness study of noisy annotation in deep learning based medical image segmentation
Authors:
- Shaode YuShaode Yu
- Mingli ChenMCMingli Chen
- Erlei ZhangEZErlei Zhang
- Junjie WuJWJunjie Wu
- Hang YuHYHang Yu
- Zi YangZYZi Yang
- Lin MaLMLin Ma
- Xuejun GuXGXuejun Gu
- Weiguo LuWLWeiguo Lu
- 21 August 2020
- journal article
- research article
- Published by IOP Publishing in Physics in Medicine & Biology
- Vol. 65 (17) , 175007
Abstract
Partly due to the use of exhaustive-annotated data, deep networks have achieved impressive performance on medical image segmentation. Medical imaging data paired with noisy annotation are, however, ubiquitous, but little is known about the effect of noisy annotation on deep learning based medical image segmentation. We studied the effect of noisy annotation in the context of mandible segmentation from CT images. First, 202 images of head and neck cancer patients were collected from our clinical database, where the organs-at-risk were annotated by one of twelve planning dosimetrists. The mandibles were roughly annotated as the planning avoiding structure. Then, mandible labels were checked and corrected by a head and neck specialist to get the reference standard. At last, by varying the ratios of noisy labels in the training set, deep networks were trained and tested for mandible segmentation. The trained models were further tested on other two public datasets. Experimental results indicated that the network trained with noisy labels had worse segmentation than that trained with reference standard, and in general, fewer noisy labels led to better performance. When using 20% or less noisy cases for training, no significant difference was found on the segmentation results between the models trained by noisy or reference annotation. Cross-dataset validation results verified that the models trained with noisy data achieved competitive performance to that trained with reference standard. This study suggests that the involved network is robust to noisy annotation to some extent in mandible segmentation from CT images. It also highlights the importance of labeling quality in deep learning. In the future work, extra attention should be paid to how to utilize a small number of reference standard samples to improve the performance of deep learning with noisy annotation.Keywords
All Related Versions
Funding Information
- Center for Scientific Review (CA235723)
This publication has 32 references indexed in Scilit:
- Automatic segmentation of the prostate on CT images using deep learning and multi-atlas fusion2017
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation2016
- Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels2016
- Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New TechniqueIEEE Transactions on Medical Imaging, 2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Metrics for evaluating 3D medical image segmentation: analysis, selection, and toolBMC Medical Imaging, 2015
- Learning from massive noisy labeled data for image classification2015
- Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection2015
- The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information RepositoryJournal of Digital Imaging, 2013
- Standardizing Naming Conventions in Radiation OncologyInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2012