Conversational AI and equity through assessing GPT-3’s communication with diverse social groups on contentious topics
Authors:
Open Access
- 18 January 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-12
Abstract
Autoregressive language models, which use deep learning to produce human-like texts, have surged in prevalence. Despite advances in these models, concerns arise about their equity across diverse populations. While AI fairness is discussed widely, metrics to measure equity in dialogue systems are lacking. This paper presents a framework, rooted in deliberative democracy and science communication studies, to evaluate equity in human–AI communication. Using it, we conducted an algorithm auditing study to examine how GPT-3 responded to different populations who vary in sociodemographic backgrounds and viewpoints on crucial science and social issues: climate change and the Black Lives Matter (BLM) movement. We analyzed 20,000 dialogues with 3290 participants differing in gender, race, education, and opinions. We found a substantively worse user experience among the opinion minority groups (e.g., climate deniers, racists) and the education minority groups; however, these groups changed attitudes toward supporting BLM and climate change efforts much more compared to other social groups after the chat. GPT-3 used more negative expressions when responding to the education and opinion minority groups. We discuss the social-technological implications of our findings for a conversational AI system that centralizes diversity, equity, and inclusion.Keywords
Funding Information
- American Family Insurance
This publication has 39 references indexed in Scilit:
- Emotion, Motivation, and the Persuasive Effects of Message FramingJournal of Communication, 2012
- The Foundations of Deliberative Democracy2012
- Modeling the acceptance of socially interactive roboticsInteraction Studies, 2011
- Climate denier, skeptic, or contrarian?Proceedings of the National Academy of Sciences, 2010
- A review of machine learning approaches to Spam filteringExpert Systems with Applications, 2009
- Understanding Technology Adoption: Theory and Future Directions for Informal LearningReview of Educational Research, 2009
- Evidence that the production of aversive consequences is not necessary to create cognitive dissonance.Journal of Personality and Social Psychology, 1996
- Aristotle on Equality and Justice1985
- The Spiral of Silence a Theory of Public OpinionJournal of Communication, 1974
- Cognitive DissonanceScientific American, 1962