Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels
Authors:
Preprint
Abstract
In this paper, we address the limitations of the common data annotation and training methods for objective single-label classification tasks. Typically, when annotating such tasks annotators are only asked to provide a single label for each sample and annotator disagreement is discarded when a final hard label is decided through majority voting. We challenge this traditional approach, acknowledging that determining the appropriate label can be difficult due to the ambiguity and lack of context in the data samples. Rather than discarding the information from such ambiguous annotations, our soft label method makes use of them for training. Our findings indicate that additional annotator information, such as confidence, secondary label and disagreement, can be used to effectively generate soft labels. Training classifiers with these soft labels then leads to improved performance and calibration on the hard label test set.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on TwitterProceedings of the International AAAI Conference on Web and Social Media, 2023
- Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective AnnotationsTransactions of the Association for Computational Linguistics, 2022
- Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks2022
- Classification aware neural topic model for COVID-19 disinformation categorisationPLOS ONE, 2021
- We Need to Consider Disagreement in Evaluation2021
- Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning2021
- Transformers: State-of-the-Art Natural Language Processing2020
- What Can We Learn from Collective Human Opinions on Natural Language Inference Data?2020
- The Benefits of a Model of AnnotationTransactions of the Association for Computational Linguistics, 2014
- Do those who know more also know more about how much they know?Organizational Behavior and Human Performance, 1977