Examining the replicability of online experiments selected by a decision market
Open Access
- 19 November 2024
- journal article
- research article
- Published by Springer Nature in Nature Human Behaviour
- Vol. 9 (2) , 316-330
Abstract
Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.Keywords
Funding Information
- Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse (P21-0091, P23-0098)
- Knut och Alice Wallenbergs Stiftelse (Wallenberg Scholar grant)
- Marianne och Marcus Wallenbergs Stiftelse, Wallenberg Scholar grant
- Riksbankens Jubileumsfond (P21-0168)
- Austrian Science Fund (SFB F63, SFB F63)
This publication has 129 references indexed in Scilit:
- Power failure: why small sample size undermines the reliability of neuroscienceNature Reviews Neuroscience, 2013
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth TellingPsychological Science, 2012
- Raise standards for preclinical cancer researchNature, 2012
- False-Positive PsychologyPsychological Science, 2011
- Why Most Discovered True Associations Are InflatedEpidemiology, 2008
- The Promise of Prediction MarketsScience, 2008
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- Statistical inference and Aristotle's RhetoricBritish Journal of Mathematical and Statistical Psychology, 2004
- A Cautionary Note on Exact Unconditional Inference for a Difference between Two Independent Binomial ProportionsBiometrics, 2003
- Significance Tests for 2 × 2 TablesBiometrika, 1947