Engaging doctoral students in peer review: a pre-post study evaluating the effectiveness of the “Peerspectives” course on review quality, knowledge and skills
Authors:
- Jessica L. RohmannJRJessica L. Rohmann
- Nadja WülkNWNadja Wülk
- Kerstin RubarthKRKerstin Rubarth
- Hannah GrillmaierHGHannah Grillmaier
- Iman AbdikarimIAIman Abdikarim
- Mariana Lopes SimoesMSMariana Lopes Simoes
- Sara SchroterSSSara Schroter
- Marco PiccininniMPMarco Piccininni
- Tobias KurthTKTobias Kurth
- Toivo GlatzTGToivo Glatz
Preprint
Abstract
Objective: To assess the effectiveness of the Peerspectives course on editor-judged quality of peer review Design: Single arm, pre-post interventional study Setting: Peerspectives is a peer review training course developed for doctoral students at the Charité – Universitätsmedizin Berlin, Germany Participants: Doctoral students in health research domains who enrolled in the Peerspectives course between October 2020 and August 2022 and consented to participate ( N = 80). Intervention: A semester-long course (approx. 18 weeks) providing training about the structure, purpose, and conduct of peer review and editing processes in biomedical journals. The course consisted of 12 hours of interactive lectures, weekly homework assignments, and 12 hours of hands-on, small-group workshops under guidance of mentors with reviewing and editing experience. Students peer-reviewed real scientific manuscripts under review at a partnering journal, The BMJ . Main outcome measures: The overall quality of the peer review reports as judged by two independent BMJ editors using the global score of the Review Quality Instrument (RQI) pre- and post-intervention. Additionally, participants’ self-assessed knowledge and skills related to scholarly peer review (1-5 Likert scale). Results: After course completion, participants’ RQI scores were higher than before the course (median increase of 0.5 points, pBMJ reviewers for the same manuscripts. Self-assessed peer review-related knowledge skills increased across all questionnaire items after course completion. Largest improvements were seen in understanding what is expected from a reviewer (increase in means from 2.9 to 4.5), confidence in own ability to review (2.5 to 3.9), and knowing what to look for in a manuscript while reviewing (2.8 to 4.2). Conclusions: Providing doctoral students with comprehensive training resulted in an editorially significant increase in review report quality and improved understanding of the role and expectations of peer reviewers in the scholarly publishing processes and confidence in giving constructive feedback. Study pre-registration: https://osf.io/vndcxKeywords
This publication has 43 references indexed in Scilit:
- Reviewer training for improving grant and journal peer reviewCochrane Database of Systematic Reviews, 2023
- How to improve scientific peer review: Four schools of thoughtLearned Publishing, 2023
- Overburdening of peer reviewers: A multi‐stakeholder perspective on causes and effectsLearned Publishing, 2021
- The nonparametric Behrens–Fisher problem in partially complete clustered dataBiometrical Journal, 2020
- What does better peer review look like? Underlying principles and recommendations for better practiceLearned Publishing, 2019
- Rewarding reviewers – sense or sensibility? A Wiley study explainedLearned Publishing, 2016
- Bias in peer reviewJournal of the American Society for Information Science and Technology, 2012
- Peer review in a changing world: An international study measuring the attitudes of researchersJournal of the American Society for Information Science and Technology, 2012
- Effect of Written Feedback by Editors on Quality of ReviewsJAMA, 2002
- The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample ApproximationBiometrical Journal, 2000