Towards Model-Based Data Acquisition for Subjective Multi-Task NLP Problems
Preprint
Abstract
Data annotated by humans is a source of knowledge by describing the peculiarities of the problem and therefore fueling the decision process of the trained model. Unfortunately, the annotation process for subjective natural language processing (NLP) problems like offensiveness or emotion detection is often very expensive and time-consuming. One of the inevitable risks is to spend some of the funds and annotator effort on annotations that do not provide any additional knowledge about the specific task. To minimize these costs, we propose a new model-based approach that allows the selection of tasks annotated individually for each text in a multi-task scenario. The experiments carried out on three datasets, dozens of NLP tasks, and thousands of annotations show that our method allows up to 40% reduction in the number of annotations with negligible loss of knowledge. The results also emphasize the need to collect a diverse amount of data required to efficiently train a model, depending on the subjectivity of the annotation task. We also focused on measuring the relation between subjective tasks by evaluating the model in single-task and multi-task scenarios. Moreover, for some datasets, training only on the labels predicted by our model improved the efficiency of task selection as a self-supervised learning regularization technique.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- ChatGPT: Jack of all trades, master of noneInformation Fusion, 2023
- Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humorInformation Fusion, 2023
- Balanced knowledge distillation for long-tailed learningNeurocomputing, 2023
- Data collection and quality challenges in deep learning: a data-centric AI perspectiveThe VLDB Journal, 2023
- Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approachInformation Processing & Management, 2021
- Directions in abusive language training data, a systematic review: Garbage in, garbage outPLOS ONE, 2020
- The SFU Opinion and Comments Corpus: A Corpus for the Analysis of Online News CommentsCorpus Pragmatics, 2019
- Garbage In, Garbage Out: Having Useful Data Is EverythingMeasurement: Interdisciplinary Research and Perspectives, 2011
- The Resident Assessment Instrument-Minimum Data Set 2.0 quality indicators: a systematic reviewBMC Health Services Research, 2010
- Methodological issues in online data collectionJournal of Advanced Nursing, 2007