IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context
Preprint
Abstract
The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India's unique socio-cultural nuances. To bridge this gap, we introduce IndiBias, a comprehensive benchmarking dataset designed specifically for evaluating social biases in the Indian context. We filter and translate the existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian context in Hindi language. Additionally, we leverage LLMs including ChatGPT and InstructGPT to augment our dataset with diverse societal biases and stereotypes prevalent in India. The included bias dimensions encompass gender, religion, caste, age, region, physical appearance, and occupation. We also build a resource to address intersectional biases along three intersectional dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias measurement across different demographics. The dataset is available in English and Hindi, providing a size comparable to existing benchmark datasets. Furthermore, using IndiBias we compare ten different language models on multiple bias measurement metrics. We observed that the language models exhibit more bias across a majority of the intersectional groups.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- mGPT: Few-Shot Learners Go MultilingualTransactions of the Association for Computational Linguistics, 2024
- Gender Bias in Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Persistent Anti-Muslim Bias in Large Language Models2021
- On the Dangers of Stochastic Parrots2021
- Re-imagining Algorithmic Fairness in India and Beyond2021
- Navigating the social world: Toward an integrated framework for evaluating self, individuals, and groups.Psychological Review, 2021
- From ‘Chinky’ to ‘Coronavirus’: racism against Northeast Indians during the Covid-19 pandemicAsian Ethnicity, 2020
- Debiasing Gender biased Hindi Words with Word-embedding2019
- Caste in the News: A Computational Analysis of Indian NewspapersSocial Media + Society, 2019
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017