Towards A Reliable Ground-Truth For Biased Language Detection
Authors:
- 1 September 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 324-325
Abstract
Reference texts such as encyclopedias and news articles can manifest biased language when objective reporting is substituted by subjective writing. Existing methods to detect bias mostly rely on annotated data to train machine learning models. However, low annotator agreement and comparability is a substantial drawback in available media bias corpora. To evaluate data collection options, we collect and compare labels obtained from two popular crowdsourcing platforms. Our results demonstrate the existing crowdsourcing approaches' lack of data quality, underlining the need for a trained expert framework to gather a more reliable dataset. By creating such a framework and gathering a first dataset, we are able to improve Krippendorff's a = 0.144 (crowdsourcing labels) to a = 0.419 (expert labels). We conclude that detailed annotator training increases data quality, improving the performance of existing bias detection systems. We will continue to extend our dataset in the future.Keywords
All Related Versions
Funding Information
- German Academic Exchange Service
This publication has 7 references indexed in Scilit:
- TASSY—A Text Annotation Survey System2021
- Do You Think It's Biased? How To Ask For The Perception Of Media Bias2021
- Automated identification of bias inducing words in news articles using linguistic and context-oriented featuresInformation Processing & Management, 2021
- Media Bias in German News Articles: A Combined Approach2020
- Neural Based Statement Classification for Biased Language2019
- Prolific.ac—A subject pool for online experimentsJournal of Behavioral and Experimental Finance, 2018
- Beyond the Turk: Alternative platforms for crowdsourcing behavioral researchJournal of Experimental Social Psychology, 2017