Revisiting Hate Speech Benchmarks: From Data Curation to System Deployment
Authors:
- Atharva KulkarniAKAtharva Kulkarni
- Sarah MasudSMSarah Masud
- Vikram GoyalVGVikram Goyal
- Tanmoy ChakrabortyTCTanmoy Chakraborty
Preprint
Abstract
Social media is awash with hateful content, much of which is often veiled with linguistic and topical diversity. The benchmark datasets used for hate speech detection do not account for such divagation as they are predominantly compiled using hate lexicons. However, capturing hate signals becomes challenging in neutrally-seeded malicious content. Thus, designing models and datasets that mimic the real-world variability of hate warrants further investigation. To this end, we present GOTHate, a large-scale code-mixed crowdsourced dataset of around 51k posts for hate speech detection from Twitter. GOTHate is neutrally seeded, encompassing different languages and topics. We conduct detailed comparisons of GOTHate with the existing hate speech datasets, highlighting its novelty. We benchmark it with 10 recent baselines. Our extensive empirical and benchmarking experiments suggest that GOTHate is hard to classify in a text-only setup. Thus, we investigate how adding endogenous signals enhances the hate speech detection task. We augment GOTHate with the user's timeline information and ego network, bringing the overall data source closer to the real-world setup for understanding hateful content. Our proposed solution HEN-mBERT is a modular, multilingual, mixture-of-experts model that enriches the linguistic subspace with latent endogenous signals from history, topology, and exemplars. HEN-mBERT transcends the best baseline by 2.5% and 5% in overall macro-F1 and hate class F1, respectively. Inspired by our experiments, in partnership with Wipro AI, we are developing a semi-automated pipeline to detect hateful content as a part of their mission to tackle online harm.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Emotionally Informed Hate Speech Detection: A Multi-target PerspectiveCognitive Computation, 2021
- Hate Speech Detection in the Bengali Language: A Dataset and Its Baseline Evaluation2021
- Hate is the New Infodemic: A Topic-aware Modeling of Hate Speech Diffusion on Twitter2021
- Resources and benchmark corpora for hate speech detection: a systematic reviewLanguage Resources and Evaluation, 2020
- Hate Speech Epidemic. The Dynamic Effects of Derogatory Language on Intergroup Relations and Political RadicalizationPolitical Psychology, 2020
- A Unified Deep Learning Architecture for Abuse Detection2019
- A Survey on Automatic Detection of Hate Speech in TextACM Computing Surveys, 2018
- Learning under Concept Drift: A ReviewIEEE Transactions on Knowledge and Data Engineering, 2018
- Deep Learning for Hate Speech Detection in Tweets2017
- node2vec2016