Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking
Authors:
- 1 April 2005
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 51 (4) , 655-661
Abstract
Peer review of research proposals and articles is an essential element in research and development processes worldwide. Here we consider a problem that, to the best of our knowledge, has not been addressed until now: how to assign subsets of proposals to reviewers in scenarios where the reviewers supply their evaluations through ordinal ranking. The solution approach we propose for this assignment problem maximizes the number of proposal pairs that will be evaluated by one or more reviewers. This new approach should facilitate meaningful aggregation of partial rankings of subsets of proposals by multiple reviewers into a consensus ranking. We offer two ways to implement the approach: an integer-programming set-covering model and a heuristic procedure. The effectiveness and efficiency of the two models are tested through an extensive simulation experiment.Keywords
This publication has 8 references indexed in Scilit:
- Peer Review in the Funding of Research in Higher Education: The Australian ExperienceEducational Evaluation and Policy Analysis, 2001
- The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review OutcomeSocial Studies of Science, 2001
- A Measure of OriginalitySocial Studies of Science, 1999
- Peer Review for Journals as it Stands Today—Part 1Science Communication, 1998
- The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigationBehavioral and Brain Sciences, 1991
- Ranking with Partial Information: A Method and an ApplicationOperations Research, 1985
- Priority Ranking and Consensus FormationManagement Science, 1978
- Preference Structures. II: Distances Between Asymmetric RelationsSIAM Journal on Applied Mathematics, 1975