Reviewer training for improving grant and journal peer review
- 28 November 2023
- journal article
- review article
- Published by Wiley in Cochrane Database of Systematic Reviews
- Vol. 2023 (11) , MR000056
Abstract
Background Funders and scientific journals use peer review to decide which projects to fund or articles to publish. Reviewer training is an intervention to improve the quality of peer review. However, studies on the effects of such training yield inconsistent results, and there are no up‐to‐date systematic reviews addressing this question. Objectives To evaluate the effect of peer reviewer training on the quality of grant and journal peer review. Search methods We used standard, extensive Cochrane search methods. The latest search date was 27 April 2022. Selection criteria We included randomized controlled trials (RCTs; including cluster‐RCTs) that evaluated peer review with training interventions versus usual processes, no training interventions, or other interventions to improve the quality of peer review. Data collection and analysis We used standard Cochrane methods. Our primary outcomes were 1. completeness of reporting and 2. peer review detection of errors. Our secondary outcomes were 1. bibliometric scores, 2. stakeholders' assessment of peer review quality, 3. inter‐reviewer agreement, 4. process‐centred outcomes, 5. peer reviewer satisfaction, and 6. completion rate and speed of funded projects. We used the first version of the Cochrane risk of bias tool to assess the risk of bias, and we used GRADE to assess the certainty of evidence. Main results We included 10 RCTs with a total of 1213 units of analysis. The unit of analysis was the individual reviewer in seven studies (722 reviewers in total), and the reviewed manuscript in three studies (491 manuscripts in total). In eight RCTs, participants were journal peer reviewers. In two studies, the participants were grant peer reviewers. The training interventions can be broadly divided into dialogue‐based interventions (interactive workshop, face‐to‐face training, mentoring) and one‐way communication (written information, video course, checklist, written feedback). Most studies were small. We found moderate‐certainty evidence that emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, have little or no effect on the completeness of reporting, measured as the proportion of items (from 0.00 to 1.00) that were adequately reported (mean difference (MD) 0.02, 95% confidence interval (CI) −0.02 to 0.06; 2 RCTs, 421 manuscripts). There was low‐certainty evidence that reviewer training, compared with standard journal practice, slightly improves peer reviewer ability to detect errors (MD 0.55, 95% CI 0.20 to 0.90; 1 RCT, 418 reviewers). We found low‐certainty evidence that reviewer training, compared with standard journal practice, has little or no effect on stakeholders' assessment of review quality in journal peer review (standardized mean difference (SMD) 0.13 standard deviations (SDs), 95% CI −0.07 to 0.33; 1 RCT, 418 reviewers), or change in stakeholders' assessment of review quality in journal peer review (SMD −0.15 SDs, 95% CI −0.39 to 0.10; 5 RCTs, 258 reviewers). We found very low‐certainty evidence that a video course, compared with no video course, has little or no effect on inter‐reviewer agreement in grant peer review (MD 0.14 points, 95% CI −0.07 to 0.35; 1 RCT, 75 reviewers). There was low‐certainty evidence that structured individual feedback on scoring, compared with general information on scoring, has little or no effect on the change in inter‐reviewer agreement in grant peer review (MD 0.18 points, 95% CI −0.14 to 0.50; 1 RCT, 41 reviewers, low‐certainty evidence). Authors' conclusions Evidence from 10 RCTs suggests that training peer reviewers may lead to little or no improvement in the quality of peer review. There is a need for studies with more participants and a broader spectrum of valid and reliable outcome measures. Studies evaluating stakeholders' assessments of the quality of peer review should ensure that these instruments have sufficient levels of validity and reliability.Keywords
This publication has 58 references indexed in Scilit:
- Bias in peer reviewJournal of the American Society for Information Science and Technology, 2012
- Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer PracticesPLOS ONE, 2012
- Does mentoring new peer reviewers improve review quality? A randomized trialBMC Medical Education, 2012
- Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trialJBI Evidence Implementation, 2011
- Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trialBMJ, 2011
- Extraneous factors in judicial decisionsProceedings of the National Academy of Sciences, 2011
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendationsBMJ, 2008
- Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized TrialPLOS ONE, 2007
- A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpointsStatistics in Medicine, 2005
- Measuring inconsistency in meta-analysesBMJ, 2003