Designing NLP Systems That Adapt to Diverse Worldviews
Preprint
Abstract
Natural Language Inference (NLI) is foundational for evaluating language understanding in AI. However, progress has plateaued, with models failing on ambiguous examples and exhibiting poor generalization. We argue that this stems from disregarding the subjective nature of meaning, which is intrinsically tied to an individual's \textit{weltanschauung} (which roughly translates to worldview). Existing NLP datasets often obscure this by aggregating labels or filtering out disagreement. We propose a perspectivist approach: building datasets that capture annotator demographics, values, and justifications for their labels. Such datasets would explicitly model diverse worldviews. Our initial experiments with a subset of the SBIC dataset demonstrate that even limited annotator metadata can improve model performance.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel BenchmarksJournal of Logic, Language and Information, 2023
- How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in TweetsProceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2023
- The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics2023
- Investigating Reasons for Disagreement in Natural Language InferenceTransactions of the Association for Computational Linguistics, 2022
- Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection2022
- Did they answer? Subjective acts and intents in conversational discourse2021
- A Case for Soft Loss FunctionsProceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2020
- Towards Robustifying NLI Models Against Lexical Dataset Biases2020
- Social Bias Frames: Reasoning about Social and Power Implications of Language2020
- Human Uncertainty Makes Classification More Robust2019