This human study did not involve human subjects: Validating LLM simulations as behavioral evidence
Preprint
Abstract
A growing literature uses large language models (LLMs) as synthetic participants to generate cost-effective and nearly instantaneous responses in social science experiments. However, there is limited guidance on when such simulations support valid inference about human behavior. We contrast two strategies for obtaining valid estimates of causal effects and clarify the assumptions under which each is suitable for exploratory versus confirmatory research. Heuristic approaches seek to establish that simulated and observed human behavior are interchangeable through prompt engineering, model fine-tuning, and other repair strategies designed to reduce LLM-induced inaccuracies. While useful for many exploratory tasks, heuristic approaches lack the formal statistical guarantees typically required for confirmatory research. In contrast, statistical calibration combines auxiliary human data with statistical adjustments to account for discrepancies between observed and simulated responses. Under explicit assumptions, statistical calibration preserves validity and provides more precise estimates of causal effects at lower cost than experiments that rely solely on human participants. Yet the potential of both approaches depends on how well LLMs approximate the relevant populations. We consider what opportunities are overlooked when researchers focus myopically on substituting LLMs for human participants in a study.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- AI Surrogates and illusions of generalizability in cognitive scienceTrends in Cognitive Sciences, 2025
- Playing repeated games with large language modelsNature Human Behaviour, 2025
- When combinations of humans and AI are useful: A systematic review and meta-analysisNature Human Behaviour, 2024
- Synthetic Replacements for Human Survey Data? The Perils of Large Language ModelsPolitical Analysis, 2024
- Artificial intelligence and illusions of understanding in scientific researchNature, 2024
- Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelinesPsychology & Marketing, 2024
- Can AI language models replace human participants?Trends in Cognitive Sciences, 2023
- Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciencesBehavioral and Brain Sciences, 2022
- The Importance of Falsification in Computational Cognitive ModelingTrends in Cognitive Sciences, 2017
- Reliability, Validity and True Values in SurveysSocial Indicators Research, 2005