How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review
Author:
Open Access
- 25 January 2024
- journal article
- review article
- Published by Springer Nature in Scientometrics
- Vol. 129 (3) , 1231-1253
Abstract
Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees’ overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most—but not all—of the referees’ judgments. Future studies should therefore examine how referees’ judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.Keywords
All Related Versions
Funding Information
- University of Zurich
This publication has 79 references indexed in Scilit:
- Simple rules for detecting depressionJournal of Applied Research in Memory and Cognition, 2013
- Bias in peer reviewJournal of the American Society for Information Science and Technology, 2012
- Heuristic Decision MakingAnnual Review of Psychology, 2011
- Editorial JudgmentsSocial Studies of Science, 2009
- Conditions for intuitive expertise: A failure to disagree.American Psychologist, 2009
- Categorization with limited resources: A family of simple heuristicsJournal of Mathematical Psychology, 2008
- Bailing and jailing the fast and frugal wayJournal of Behavioral Decision Making, 2001
- Judgment under Uncertainty: Heuristics and BiasesScience, 1974
- The use of nonlinear, noncompensatory models in decision making.Psychological Bulletin, 1970
- Clinical versus Statistical Prediction: A Theoretical Analysis and a Review of the EvidenceThe American Journal of Psychology, 1955