Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study
- 9 July 2016
- journal article
- research article
- Published by SAGE Publications in Comparative Political Studies
- Vol. 49 (13) , 1667-1703
Abstract
In 2015, Comparative Political Studies embarked on a landmark pilot study in research transparency in the social sciences. The editors issued an open call for submissions of manuscripts that contained no mention of their actual results, incentivizing reviewers to evaluate manuscripts based on their theoretical contributions, research designs, and analysis plans. The three papers in this special issue are the result of this process that began with 19 submissions. In this article, we describe the rationale for this pilot, expressly articulating the practices of preregistration and results-free review. We document the process of carrying out the special issue with a discussion of the three accepted papers, and critically evaluate the role of both preregistration and results-free review. Our main conclusions are that results-free review encourages much greater attention to theory and research design, but that it raises thorny problems about how to anticipate and interpret null findings. We also observe that as currently practiced, results-free review has a particular affinity with experimental and cross-case methodologies. Our lack of submissions from scholars using qualitative or interpretivist research suggests limitations to the widespread use of results-free review.Keywords
This publication has 41 references indexed in Scilit:
- Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*The Quarterly Journal of Economics, 2012
- A Model Discipline2012
- False-Positive PsychologyPsychological Science, 2011
- Publication Bias in Two Political Behavior LiteraturesAmerican Politics Research, 2010
- Growth in a Time of DebtAmerican Economic Review, 2010
- Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science JournalsQuarterly Journal of Political Science, 2008
- The Effects of U.S. Foreign Assistance on Democracy Building, 1990–2003World Politics, 2007
- Nested Analysis as a Mixed-Method Strategy for Comparative ResearchAmerican Political Science Review, 2005
- The Employment Effects of Minimum Wages: Evidence from a Prespecified Research Design The Employment Effects of MinimumWagesIndustrial Relations: A Journal of Economy and Society, 2001
- The Insignificance of Null Hypothesis Significance TestingPolitical Research Quarterly, 1999