Are Human and LLM-Generated Observations Statistically Comparable? Rethinking Sampling Units, Variability, and Statistical Inference
Open Access
Abstract
The increasing use of Large Language Models (LLMs) as synthetic respondents, experimental participants, or comparators to human samples and populations raises a critical statistical question: under what conditions can LLM-generated observations support valid inductive reasoning alongside human data? This paper distinguishes between the heterogeneity inherent in human populations and the algorithmic variability of LLMs, emphasising that similarity in observed responses does not necessarily indicate equivalence in the underlying generative processes. Human data are derived from individuals sampled from a target population. In contrast, repeated LLM responses are generated by a probabilistic computational system, with variability influenced by the model, prompt, production parameters, context, repeated runs, and computational environment. Consequently, the nominal number of LLM-generated responses should not be equated with an equivalent number of independent human sampling units. This distinction affects the independence of observations, repeatability, sample size, statistical testing, and the interpretation of human–LLM comparisons. Conventional statistical methods are not inherently unsuitable for LLM-generated data; however, their validity depends on whether the experimental unit, estimand, dependency structure, and methodological assumptions align with the data-generating process. To support practical application, a six-stage methodological framework is proposed: defining the inference goal, identifying units and data-generation processes, assessing replication, validating human–LLM comparability, selecting appropriate statistical analyses, and evaluating the sensitivity, reproducibility, and scope of inference. Human–LLM comparability should be treated as an empirical, estimand-specific property rather than assumed from equal sample sizes, similar means, non-significant differences, or the application of identical statistical tests.Keywords
This publication has 46 references indexed in Scilit:
- Recent trends of machine learning techniques for risk assessment in hazardous environmentsArtificial Intelligence Review, 2026
- From hallucinations to hazards: benchmarking LLMs for hazard analysis in safety-critical systemsSafety Science, 2025
- Are answers obtained from artificial intelligence models for information purposes repeatable?International Orthodontics, 2025
- Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?Journal of Systems and Software, 2025
- Participant Interactions with Artificial Intelligence: Using Large Language Models to Generate Research Materials for Surveys and ExperimentsJournal of Business and Psychology, 2025
- AI Computing Systems for Large Language Models TrainingJournal of Computer Science and Technology, 2025
- History, development, and principles of large language models: an introductory surveyAI and Ethics, 2024
- Comparing advice on climate policy between academic experts and ChatGPTEcological Economics, 2024
- Hazard analysis in the era of AI: Assessing the usefulness of ChatGPT4 in STPA hazard analysisSafety Science, 2024
- Evaluating the Application of Large Language Models in Clinical Research ContextsJAMA Network Open, 2023