Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Authors:
Top Cited Papers
Open Access
- 4 September 2024
- journal article
- research article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 12, 1011-1026
Abstract
One widely cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording—but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of “prompts” have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF. Furthermore, even if a model shows a significant change in the same direction as humans, we find that they are sensitive to perturbations that do not elicit significant changes in humans. These results highlight the pitfalls of using LLMs as human proxies, and underscore the need for finer-grained characterizations of model behavior.1Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- Can AI language models replace human participants?Trends in Cognitive Sciences, 2023
- Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study2023
- Social Simulacra: Creating Populated Prototypes for Social Computing Systems2022
- The Significance of Letter Position in Word RecognitionIEEE Aerospace and Electronic Systems Magazine, 2007
- Do Polls Reflect Opinions or Do Opinions Reflect Polls? The Impact of Political Polling on Voters' Expectations, Preferences, and BehaviorJournal of Consumer Research, 1996
- A Cognitive Model of Response-Order Effects in Survey Measurement1992
- Response Effects in Mail SurveysPublic Opinion Quarterly, 1990
- Response Effects in Surveys1987
- Effects of Question Order on Survey ResponsesPublic Opinion Quarterly, 1981