Representativeness and response validity across nine opt-in online samples
Top Cited Papers
Open Access
- 22 June 2026
- journal article
- Published by Springer Nature in Nature Human Behaviour
- Vol. 10 (8) , 1441-1454
Abstract
The study of human behaviour has shifted in the past 15 years. It increasingly relies on opt-in non-probability online data sources. Here we offer an analysis of nine such sources (N = 13,053), aiming to inform researchers conducting experiments or correlational studies. We assess response validity (attentiveness, effort, honesty, speeding and attrition), the extent to which samples represent the underlying population (demographics, attitude representativeness and experimental treatment effects) and professionalism (number and frequency of studies taken and modality of device on which studies are taken). We document substantial variation across samples on each dimension. Samples using demographic quotas display relatively higher amounts of representativeness across multiple indicators (beyond demographics) but often exhibit less response validity. However, inclusion of two attention checks early in a survey can enhance response validity without negatively impacting representativeness. We offer guidance for choosing opt-in samples, depending on the purpose of the research and resource constraints.Keywords
This publication has 61 references indexed in Scilit:
- A synthesis of evidence for policy from behavioural science during COVID-19Nature, 2023
- Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONAPLOS ONE, 2023
- Protecting the integrity of survey researchPNAS Nexus, 2023
- The day-of-invitation effect on participation in web-based studiesBehavior Research Methods, 2021
- Data quality of platforms and panels for online behavioral researchBehavior Research Methods, 2021
- Behavioural science is unlikely to change the world without a heterogeneity revolutionNature Human Behaviour, 2021
- Turking in the time of COVIDBehavior Research Methods, 2021
- Generalizability of heterogeneous treatment effect estimates across samplesProceedings of the National Academy of Sciences, 2018
- Social heuristics shape intuitive cooperationNature Communications, 2014
- The Influence of Framing on Risky Decisions: A Meta-analysisOrganizational Behavior and Human Decision Processes, 1998