To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction
Authors:
Preprint
Abstract
This paper explores the task of automatic prediction of text spans in a legal problem description that support a legal area label. We use a corpus of problem descriptions written by laypeople in English that is annotated by practising lawyers. Inherent subjectivity exists in our task because legal area categorisation is a complex task, and lawyers often have different views on a problem, especially in the face of legally-imprecise descriptions of issues. Experiments show that training on majority-voted spans outperforms training on disaggregated ones.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)2023
- The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation2022
- Semi-automatic Triage of Requests for Free Legal Assistance2021
- Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning2021
- Weakly- and Semi-supervised Evidence Extraction2020
- What Can We Learn from Collective Human Opinions on Natural Language Inference Data?2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Neural Architectures for Named Entity Recognition2016
- Get another label? improving data quality and data mining using multiple, noisy labelers2008
- Long Short-Term MemoryNeural Computation, 1997