Pre-Analysis Plans: An Early Stocktaking
Open Access
- 31 March 2021
- journal article
- research article
- Published by Cambridge University Press (CUP) in Perspectives on Politics
- Vol. 21 (1) , 174-190
Abstract
Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze a representative sample of 195 PAPs registered on the Evidence in Governance and Politics (EGAP) and American Economic Association (AEA) registration platforms to assess whether PAPs registered in the early days of pre-registration (2011–2016) were sufficiently clear, precise, and comprehensive to achieve their objective of preventing “fishing” and reducing the scope for post-hoc adjustment of research hypotheses. We also analyze a subset of ninety-three PAPs from projects that resulted in publicly available papers to ascertain how faithfully they adhere to their pre-registered specifications and hypotheses. We find significant variation in the extent to which PAPs registered during this period accomplished the goals they were designed to achieve. We discuss these findings in light of both the costs and benefits of pre-registration, showing how our results speak to the various arguments that have been made in support of and against PAPs. We also highlight the norms and institutions that will need to be strengthened to augment the power of PAPs to improve research credibility and to create incentives for researchers to invest in both producing and policing them.Keywords
This publication has 31 references indexed in Scilit:
- Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are FeasibleJournal of Economic Perspectives, 2015
- Publication bias in the social sciences: Unlocking the file drawerScience, 2014
- P-curve: A key to the file-drawer.Journal of Experimental Psychology: General, 2014
- Fisheries ManagementPolitical Analysis, 2013
- Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research RegistrationPolitical Analysis, 2013
- Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*The Quarterly Journal of Economics, 2012
- False-Positive PsychologyPsychological Science, 2011
- Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science JournalsQuarterly Journal of Political Science, 2008
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- The file drawer problem and tolerance for null results.Psychological Bulletin, 1979