Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Preprint
Abstract
Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Overview of Touché 2024: Argumentation Systems2024
- How context shapes value co-creation: spectator experience of sport eventsThe Service Industries Journal, 2016
- Institutions and axioms: an extension and update of service-dominant logicJournal of the Academy of Marketing Science, 2015
- Refining the theory of basic individual values.Journal of Personality and Social Psychology, 2012
- The Study of Human Nature and the Subjectivity of Value2011
- Expanding understanding of service exchange and value co-creation: a social construction approachJournal of the Academy of Marketing Science, 2010
- When Morality Opposes Justice: Conservatives Have Moral Intuitions that Liberals may not RecognizeSocial Justice Research, 2007
- Reliability in Content Analysis: Some Common Misconceptions and RecommendationsHuman Communication Research, 2004
- Persuasion in Practical Argument Using Value-based Argumentation FrameworksJournal of Logic and Computation, 2003
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971