Examining Temporal Bias in Abusive Language Detection
Preprint
Abstract
The use of abusive language online has become an increasingly pervasive problem that damages both individuals and society, with effects ranging from psychological harm right through to escalation to real-life violence and even death. Machine learning models have been developed to automatically detect abusive language, but these models can suffer from temporal bias, the phenomenon in which topics, language use or social norms change over time. This study aims to investigate the nature and impact of temporal bias in abusive language detection across various languages and explore mitigation methods. We evaluate the performance of models on abusive data sets from different time periods. Our results demonstrate that temporal bias is a significant challenge for abusive language detection, with models trained on historical data showing a significant drop in performance over time. We also present an extensive linguistic analysis of these abusive data sets from a diachronic perspective, aiming to explore the reasons for language evolution and performance decline. This study sheds light on the pervasive issue of temporal bias in abusive language detection across languages, offering crucial insights into language evolution and temporal bias mitigation.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Building for tomorrow: Assessing the temporal persistence of text classifiersInformation Processing & Management, 2022
- Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting2022
- SWSR: A Chinese dataset and lexicon for online sexism detectionOnline Social Networks and Media, 2022
- Abusive Bangla comments detection on Facebook using transformer-based deep learning modelsSocial Network Analysis and Mining, 2021
- Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness MetricsTransactions of the Association for Computational Linguistics, 2021
- Five sources of bias in natural language processingLanguage and Linguistics Compass, 2021
- Bias and comparison framework for abusive language datasetsAI and Ethics, 2021
- Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement2021
- Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection2021
- Data preprocessing techniques for classification without discriminationKnowledge and Information Systems, 2011