Human labeling errors and their impact on ConvNets for satellite image scene classification
Authors:
Preprint
Abstract
Convolutional neural networks (ConvNets) have been successfully applied to satellite image scene classification. Human-labeled training datasets are essential for ConvNets to perform accurate classification. Errors in human-labeled training datasets are unavoidable due to the complexity of satellite images. However, the distribution of human labeling errors on satellite images and their impact on ConvNets have not been investigated. To fill this research gap, this study, for the first time, collected real-world labels from 32 participants and explored how their errors affect three ConvNets (VGG16, GoogleNet and ResNet-50) for high-resolution satellite image scene classification. We found that: (1) human labeling errors have significant class and instance dependence, which is fundamentally different from the simulation noise in previous studies; (2) regarding the overall accuracy of all classes, when human labeling errors in training data increase by one unit, the overall accuracy of ConvNets classification decreases by approximately half a unit; (3) regarding the accuracy of each class, the impact of human labeling errors on ConvNets shows large heterogeneity across classes. To uncover the mechanism underlying the impact of human labeling errors on ConvNets, we further compared it with two types of simulated labeling noise: uniform noise (errors independent of both classes and instances) and class-dependent noise (errors independent of instances but not classes). Our results show that the impact of human labeling errors on ConvNets is similar to that of the simulated class-dependent noise but not to that of the simulated uniform noise, suggesting that the impact of human labeling errors on ConvNets is mainly due to class-dependent errors rather than instance-dependent errors.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Current trends in deep learning for Earth Observation: An open-source benchmark arena for image classificationISPRS Journal of Photogrammetry and Remote Sensing, 2023
- SceneNet: Remote sensing scene classification deep learning network using multi-objective neural evolution architecture searchISPRS Journal of Photogrammetry and Remote Sensing, 2021
- A survey of the recent architectures of deep convolutional neural networksArtificial Intelligence Review, 2020
- Towards better exploiting convolutional neural networks for remote sensing scene classificationPattern Recognition, 2017
- Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection2015
- Multi-class geospatial object detection and geographic image classification based on collection of part detectorsISPRS Journal of Photogrammetry and Remote Sensing, 2014
- Supervised Learning2011
- A study of the effect of different types of noise on the precision of supervised learning techniquesArtificial Intelligence Review, 2010
- Class Noise vs. Attribute Noise: A Quantitative StudyArtificial Intelligence Review, 2004
- Selecting and interpreting measures of thematic classification accuracyRemote Sensing of Environment, 1997