Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions
Preprint
Abstract
As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs are susceptible to societal biases due to their exposure to human-generated data. Given that LLMs are being used to gain insights into various societal aspects, it is essential to mitigate these biases. To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases. We begin by creating a dataset of scenarios where implicit gender biases might arise, and subsequently develop a metric to assess the presence of biases. Our empirical analysis reveals that LLMs generate outputs characterized by strong implicit bias associations (>= 50\% of the time). Furthermore, these biases tend to escalate following multi-agent interactions. To mitigate them, we propose two strategies: self-reflection with in-context examples (ICE); and supervised fine-tuning. Our research demonstrates that both methods effectively mitigate implicit biases, with the ensemble of fine-tuning and self-reflection proving to be the most successful.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching RegularizationJournal of the American Statistical Association, 2025
- Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge2024
- Small Language Model Can Self-CorrectProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Association between physical activity, sedentary time, participation in organized activities, social support, sleep problems and mental distress among adults in Southern Norway: a cross-sectional study among 28,047 adults from the general populationBMC Public Health, 2022
- On the Dangers of Stochastic Parrots2021
- Gender Bias in Neural Natural Language Processing2020
- Mind the GAP: A Balanced Corpus of Gendered Ambiguous PronounsTransactions of the Association for Computational Linguistics, 2018
- Technical boys and creative girls: the career aspirations of digitally skilled youthsCambridge Journal of Education, 2017
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017
- Investors prefer entrepreneurial ventures pitched by attractive menProceedings of the National Academy of Sciences, 2014