Tracking Replicability as a Method of Post-Publication Open Evaluation
Open Access
- 1 January 2012
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Computational Neuroscience
- Vol. 6, 11987
Abstract
Recent reports have suggested that many published results are unreliable. To increase the reliability and accuracy of published papers, multiple changes have been proposed, such as changes in statistical methods. We support such reforms. However, we believe that the incentive structure of scientific publishing must change for such reforms to be successful. Under the current system, the quality of individual scientists is judged on the basis of their number of publications and citations, with journals similarly judged via numbers of citations. Neither of these measures takes into account the replicability of the published findings, as false or controversial results are often particularly widely cited. We propose tracking replications as a means of post-publication evaluation, both to help researchers identify reliable findings and to incentivize the publication of reliable results. Tracking replications requires a database linking published studies that replicate one another. As any such database is limited by the number of replication attempts published, we propose establishing an open-access journal dedicated to publishing replication attempts. Data quality of both the database and the affiliated journal would be ensured through a combination of crowd-sourcing and peer review. As reports in the database are aggregated, ultimately it will be possible to calculate replicability scores, which may be used alongside citation counts to evaluate the quality of work published in individual journals. In this paper, we lay out a detailed description of how this system could be implemented, including mechanisms for compiling the information, ensuring data quality, and incentivizing the research community to participate.Keywords
This publication has 72 references indexed in Scilit:
- Samples in applied psychology: Over a decade of research in review.Journal of Applied Psychology, 2011
- Efficacy of cognitive–behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication biasThe British Journal of Psychiatry, 2010
- Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social CognitionPerspectives on Psychological Science, 2009
- Bias in genetic association studies and impact factorMolecular Psychiatry, 2009
- False-Positive Results in Cancer Epidemiology: A Plea for Epistemological ModestyJNCI: Journal of the National Cancer Institute, 2008
- The fading of reported effectiveness. A meta-analysis of randomised controlled trialsBMC Medical Research Methodology, 2006
- Genetic associations in large versus small studies: an empirical assessmentThe Lancet, 2003
- Publication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ methodBiological Reviews, 2002
- A comprehensive review of genetic association studiesGenetics in Medicine, 2002
- Epidemiology Faces Its Limits1995