NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing
Preprint
Abstract
Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade model generalization. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance. Due to the lack of suitable datasets, previous studies have frequently employed synthetic label noise to mimic real-world label noise. However, synthetic noise is not instance-dependent, making this approximation not always effective in practice. Recent research has proposed benchmarks for learning with real-world noisy labels. However, the noise sources within may be single or fuzzy, making benchmarks different from data with heterogeneous label noises in the real world. To tackle these issues, we contribute NoisywikiHow, the largest NLP benchmark built with minimal supervision. Specifically, inspired by human cognition, we explicitly construct multiple sources of label noise to imitate human errors throughout the annotation, replicating real-world noise, whose corruption is affected by both ground-truth labels and instances. Moreover, we provide a variety of noise levels to support controlled experiments on noisy data, enabling us to evaluate LNL methods systematically and comprehensively. After that, we conduct extensive multi-dimensional experiments on a broad range of LNL methods, obtaining new and intriguing findings.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- DiscrimLoss: A Universal Loss for Hard Samples and Incorrect Samples DiscriminationIEEE Transactions on Multimedia, 2023
- Event Linking: Grounding Event Mentions to Wikipedia2023
- STEPS: Semantic Typing of Event Processes with a Sequence-to-Sequence ApproachProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Learning From Noisy Labels With Deep Neural Networks: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2022
- Audio Tagging with Noisy Labels and Minimal Supervision2019
- ConceptNet — A Practical Commonsense Reasoning Tool-KitBT Technology Journal, 2004