How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets
Top Cited Papers
Open Access
- 3 November 2023
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Vol. 11 (1) , 38-50
Abstract
Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions. However, annotation tasks such as judging hate speech are subjective and thus highly sensitive to biases stemming from annotator beliefs, characteristics and demographics. We conduct two crowdsourcing studies on Mechanical Turk to examine annotator bias in labelling sexist and misogynistic hate speech. Results from 109 annotators show that annotator political inclination, moral integrity, personality traits, and sexist attitudes significantly impact annotation accuracy and the tendency to tag content as hate speech. In addition, semi-structured interviews with nine crowd workers provide further insights regarding the influence of subjectivity on annotations. In exploring how workers interpret a task — shaped by complex negotiations between platform structures, task instructions, subjective motivations, and external contextual factors — we see annotations not only impacted by worker factors but also simultaneously shaped by the structures under which they labour.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity AnnotationProceedings of the ACM on Human-Computer Interaction, 2022
- Crowdwork, digital liminality and the enactment of culturally recognised alternatives to Western precarity: beyond epistemological terra nulliusEuropean Journal of Information Systems, 2021
- Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approachInformation Processing & Management, 2021
- Crowdsourcing as a Tool for Research: Methodological, Fair, and Political ConsiderationsBulletin of Science, Technology & Society, 2020
- Resources and benchmark corpora for hate speech detection: a systematic reviewLanguage Resources and Evaluation, 2020
- What Do Platforms Do? Understanding the Gig EconomyAnnual Review of Sociology, 2020
- Online Hate Interpretation Varies by Country, But More by Individual: A Statistical Analysis Using Crowdsourced Ratings2018
- A Survey on Automatic Detection of Hate Speech in TextACM Computing Surveys, 2018
- Development, reliability, and validity of the Moral Identity QuestionnairePersonality and Individual Differences, 2016
- Answering the Call for a Standard Reliability Measure for Coding DataCommunication Methods and Measures, 2007