Madhuri Sawant

The effectiveness of supervised machine learning models is heavily influenced by the quality of training data, which is often shaped by human annotators. Subjective NLP tasks such as hate speech detection, toxicity identification, and sexism classification frequently exhibit annotator disagreement due to differences in individual perspectives. This study investigates annotator disagreement in sexism detection using English tweets from the EXIST 2023 competition. To systematically analyse disagreement, tweets are categorised based on annotator consensus levels, examining how annotator demographics and linguistic features contribute to labelling inconsistencies. We interpret disagreement patterns using Shapley Additive Explanations (SHAP) and assess the consistency of SHAP-derived feature importance rankings via Spearman Rank Correlation. Our findings demonstrate that both annotator demographics and tweet characteristics significantly shape disagreement, reinforcing the need for perspectivist approaches in NLP by showing that annotator disagreement is not just noise but a meaningful signal that should be incorporated into dataset construction. Please be advised that this work contains examples of offensive content

The proliferation of hate speech on digital platforms has become a significant issue, and automated content moderation systems built on machine learning are a proposed solution. However, they face challenges in multilingual and low-resource settings due to the need for extensive labelled data. This paper introduces an explainable AI framework designed to identify annotation discrepancies in low-resource languages, focusing on Hindi, the third most-spoken language worldwide, which lacks comprehensive research in hate speech detection. By examining the labelling quality of the Hate speech and Offensive Content Identification in English and Indo-Aryan Languages (HASOC) challenge, we use unsupervised learning methods to extract topical variations and annotation behavior and apply these features in an explainable AI-based classification model, TabNet. We release a relabelled Hindi hate speech benchmark dataset with label-flipping information and related metadata to facilitate research in this area. The source code has also been released for reproducibility purposes. Please be advised that this work contains examples of toxic content