Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation
Authors:
- Jizhi ZhangJZJizhi Zhang
- Keqin BaoKBKeqin Bao
- Yang ZhangYZYang Zhang
- Wenjie WangWWWenjie Wang
- Fuli FengFFFuli Feng
- Xiangnan HeXHXiangnan He
Preprint
Abstract
The remarkable achievements of Large Language Models (LLMs) have led to the emergence of a novel recommendation paradigm -- Recommendation via LLM (RecLLM). Nevertheless, it is important to note that LLMs may contain social prejudices, and therefore, the fairness of recommendations made by RecLLM requires further investigation. To avoid the potential risks of RecLLM, it is imperative to evaluate the fairness of RecLLM with respect to various sensitive attributes on the user side. Due to the differences between the RecLLM paradigm and the traditional recommendation paradigm, it is problematic to directly use the fairness benchmark of traditional recommendation. To address the dilemma, we propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM). This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes1 in two recommendation scenarios: music and movies. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations. Our code and dataset can be found at https://github.com/jizhi-zhang/FaiRLLM.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation2023
- A Survey on the Fairness of Recommender SystemsACM Transactions on Information Systems, 2023
- An Audit of Misinformation Filter Bubbles on YouTube: Bubble Bursting and Recent Behavior Changes2021
- Large language models associate Muslims with violenceNature Machine Intelligence, 2021
- Leave No User Behind: Towards Improving the Utility of Recommender Systems for Non-mainstream Users2021
- On the negative impact of social influence in recommender systems: A study of bribery in collaborative hybrid algorithmsInformation Processing & Management, 2020
- Surveying Multiple Sensitive Attributes using an Extension of the Randomized-Response TechniqueInternational Journal of Public Opinion Research, 2011