Large language models display human-like social desirability biases in Big Five personality surveys
Top Cited Papers
Open Access
- 28 November 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in PNAS Nexus
- Vol. 3 (12) , pgae533
Abstract
Large language models (LLMs) are becoming more widely used to simulate human participants and so understanding their biases is important. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e. increased extraversion, decreased neuroticism, etc.). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4’s survey responses changing by 1.20 (human) SD and Llama 3’s by 0.98 SD, which are very large effects. This bias remains after question order randomization and paraphrasing. Reverse coding the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on this use of LLMs as proxies for human participants.Keywords
Funding Information
- Stanford Institute for Human-Centered AI
- Artificial Intelligence in Medicine
- Imaging at Stanford University
This publication has 14 references indexed in Scilit:
- A Turing test of whether AI chatbots are behaviorally similar to humansProceedings of the National Academy of Sciences of the United States of America, 2024
- Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response biasProceedings of the National Academy of Sciences, 2023
- Can AI language models replace human participants?Trends in Cognitive Sciences, 2023
- Out of One, Many: Using Language Models to Simulate Human SamplesPolitical Analysis, 2023
- Using cognitive psychology to understand GPT-3Proceedings of the National Academy of Sciences, 2023
- To Reverse Item Orientation or Not to Reverse Item Orientation, That Is the QuestionAssessment, 2021
- The Power of Personality: The Comparative Validity of Personality Traits, Socioeconomic Status, and Cognitive Ability for Predicting Important Life OutcomesPerspectives on Psychological Science, 2007
- Social Desirability and Self-Reports: Testing Models of Socially Desirable RespondingPersonality and Social Psychology Bulletin, 2004
- Self-reports: How the questions shape the answers.American Psychologist, 1999
- Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin, 1959