Retrospective median power, false positive meta‐analysis and large‐scale replication
- 10 October 2021
- journal article
- research article
- Published by Wiley in Research Synthesis Methods
- Vol. 13 (1) , 88-108
Abstract
Recent, high‐profile, large‐scale, preregistered failures to replicate uncover that many highly‐regarded experiments are “false positives”; that is, statistically significant results of underlying null effects. Large surveys of research reveal that statistical power is often low and inadequate. When the research record includes selective reporting, publication bias and/or questionable research practices, conventional meta‐analyses are also likely to be falsely positive. At the core of research credibility lies the relation of statistical power to the rate of false positives. This study finds that high (>50%–60%) median retrospective power (MRP) is associated with credible meta‐analysis and large‐scale, preregistered, multi‐lab “successful” replications; that is, with replications that corroborate the effect in question. When median retrospective power is low (<50%), positive meta‐analysis findings should be interpreted with great caution or discounted altogether.Keywords
This publication has 76 references indexed in Scilit:
- Meta-regression approximations to reduce publication selection biasResearch Synthesis Methods, 2013
- The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane ReviewsPLOS ONE, 2013
- Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.Journal of Personality and Social Psychology, 2011
- Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation studyBMC Medical Research Methodology, 2009
- Incidental environmental anchorsJournal of Behavioral Decision Making, 2007
- Selection in Reported Epidemiological Risks: An Empirical AssessmentPLOS Medicine, 2007
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- One Hundred Years of Social Psychology Quantitatively DescribedReview of General Psychology, 2003
- The power of statistical tests in meta-analysis.Psychological Methods, 2001
- A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-AnalysisJournal of the American Statistical Association, 2000