Human-annotated label noise and their impact on ConvNets for remote sensing image scene classification
Preprint
Abstract
Convolutional neural networks (ConvNets) have been successfully applied to satellite image scene classification. Human-labeled training datasets are essential for ConvNets to perform accurate classification. Errors in human-annotated training datasets are unavoidable due to the complexity of satellite images. However, the distribution of real-world human-annotated label noises on remote sensing images and their impact on ConvNets have not been investigated. To fill this research gap, this study, for the first time, collected real-world labels from 32 participants and explored how their annotated label noise affect three representative ConvNets (VGG16, GoogleNet, and ResNet-50) for remote sensing image scene classification. We found that: (1) human-annotated label noise exhibits significant class and instance dependence; (2) an additional 1% of human-annotated label noise in training data leads to 0.5% reduction in the overall accuracy of ConvNets classification; (3) the error pattern of ConvNet predictions was strongly correlated with that of participant's labels. To uncover the mechanism underlying the impact of human labeling errors on ConvNets, we further compared it with three types of simulated label noise: uniform noise, class-dependent noise and instance-dependent noise. Our results show that the impact of human-annotated label noise on ConvNets significantly differs from all three types of simulated label noise, while both class dependence and instance dependence contribute to the impact of human-annotated label noise on ConvNets. These observations necessitate a reevaluation of the handling of noisy labels, and we anticipate that our real-world label noise dataset would facilitate the future development and assessment of label-noise learning algorithms.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Mapping local climate zones for cities: A large reviewRemote Sensing of Environment, 2023
- Current trends in deep learning for Earth Observation: An open-source benchmark arena for image classificationISPRS Journal of Photogrammetry and Remote Sensing, 2023
- Urban scene understanding based on semantic and socioeconomic features: From high-resolution remote sensing imagery to multi-source geographic datasetsISPRS Journal of Photogrammetry and Remote Sensing, 2021
- SceneNet: Remote sensing scene classification deep learning network using multi-objective neural evolution architecture searchISPRS Journal of Photogrammetry and Remote Sensing, 2021
- Adopting deep learning methods for airborne RGB fluvial scene classificationRemote Sensing of Environment, 2020
- Towards better exploiting convolutional neural networks for remote sensing scene classificationPattern Recognition, 2017
- Multi-class geospatial object detection and geographic image classification based on collection of part detectorsISPRS Journal of Photogrammetry and Remote Sensing, 2014
- A study of the effect of different types of noise on the precision of supervised learning techniquesArtificial Intelligence Review, 2010
- Class Noise vs. Attribute Noise: A Quantitative StudyArtificial Intelligence Review, 2004
- Learning from noisy examplesMachine Learning, 1988