Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review
- 21 February 2012
- journal article
- Published by Springer Nature in Surgical Endoscopy
- Vol. 26 (8) , 2275-2280
Abstract
The peer review process is the gold standard by which academic manuscripts are vetted for publication. However, some investigators have raised concerns regarding its unopposed supremacy, including lack of expediency, susceptibility to editorial bias and statistical limitation due to the small number of reviewers used. Post-publication review—in which the article is assessed by the general readership of the journal instead of a small group of appointed reviewers—could potentially supplement or replace the peer-review process. In this study, we created a computer model to compare the traditional peer-review process to that of post-publication reader review. We created a mathematical model of the manuscript review process. A hypothetical manuscript was randomly assigned a “true value” representing its intrinsic quality. We modeled a group of three expert peer reviewers and compared it to modeled groups of 10, 20, 50, or 100 reader-reviewers. Reader-reviewers were assumed to be less skillful at reviewing and were thus modeled to be only ¼ as accurate as expert reviewers. Percentage of correct assessments was calculated for each group. 400,000 hypothetical manuscripts were modeled. The accuracy of the reader-reviewer group was inferior to the expert reviewer group in the 10-reviewer trial (93.24% correct vs. 97.67%, p < 0.0001) and the 20-reviewer trial (95.50% correct, p < 0.0001). However, the reader-reviewer group surpassed the expert reviewer group in accuracy when 50 or 100 reader-reviewers were used (97.92 and 99.20% respectively, p < 0.0001). In a mathematical model of the peer review process, the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size. Further study will be required to determine whether the mathematical assumptions of this model are valid in actual use.Keywords
This publication has 11 references indexed in Scilit:
- Peer review: Trial by TwitterNature, 2011
- Do Author-Suggested Reviewers Rate Submissions More Favorably than Editor-Suggested Reviewers? A Study on Atmospheric Chemistry and PhysicsPLOS ONE, 2010
- What errors do peer reviewers detect, and does training improve their ability to detect them?Journal of the Royal Society of Medicine, 2008
- The Effectiveness of the Peer Review Process: Inter‐Referee Agreement and Predictive Validity of Manuscript Refereeing at Angewandte ChemieAngewandte Chemie International Edition, 2008
- Editorial peer review for improving the quality of reports of biomedical studiesCochrane Database of Systematic Reviews, 2007
- Effects of Editorial Peer ReviewJAMA, 2002
- Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?Brain, 2000
- Chance and Consensus in Peer ReviewScience, 1981
- Vox PopuliNature, 1907
- The Ballot-BoxNature, 1907