Automating Bias Testing of LLMs
- 11 September 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1705-1707
Abstract
Large Language Models (LLMs) are being quickly integrated in a myriad of software applications. This may introduce a number of biases, such as gender, age or ethnicity, in the behavior of such applications. To face this challenge, we explore the automatic generation of tests suites to assess the potential biases of an LLM. Each test is defined as a prompt used as input to the LLM and a test oracle that analyses the LLM output to detect the presence of biases.Keywords
This publication has 13 references indexed in Scilit:
- Using Natural Sentence Prompts for Understanding Biases in Language Models2022
- Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPTransactions of the Association for Computational Linguistics, 2021
- Operationalising AI ethics: barriers, enablers and next stepsAI & SOCIETY, 2021
- Operationalising AI ethics: how are companies bridging the gap between practice and principles? An exploratory studyAI & SOCIETY, 2021
- Investigating Gender Bias in BERTCognitive Computation, 2021
- BOLD2021
- The Ethics of AI Ethics: An Evaluation of GuidelinesMinds and Machines, 2020
- The Woman Worked as a Babysitter: On Biases in Language Generation2019
- Evaluating the Underlying Gender Bias in Contextualized Word Embeddings2019
- Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods2018