Improving the efficiency of research proposals evaluation: A two-stage procedure
Open Access
- 15 May 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in Research Evaluation
Abstract
An important share of research funding is allocated via competitive programs, which entail considerable direct and indirect costs, such as to develop and evaluate the proposals. The goal of this article is to explore whether adopting a two-stage evaluation procedure could improve the efficiency of the process. For this purpose, we study the evaluation system designed by the Foundation Dam (Stiftelsen Dam), one of the largest foundations in Norway supporting health research. In 2020, Foundation Dam adopted a new evaluation procedure consisting in a short proposal for a first stage of selection and, for those selected, a second-stage evaluation of a long proposal. We explore whether such a procedure reduces the evaluation costs and how the evaluation procedures compare in terms of reliability. Survey responses from 94 of the 594 applicants in the one-stage procedure (2018–19) and all the 668 applicants in the two-stage procedure (2020–21) show that the two-stage procedure reduced the average time that applicants spent in drafting the proposal(s) by 38%. According to the reviewers’ estimate, the two-stage procedure also reduced by 28% the time they spent on average to evaluate an applicant’s proposal(s). The analysis of the evaluation data of 594 long proposals in the one-stage procedure and 668 short proposals in the two-stage procedure shows that reviewers’ scores of short proposals display greater reliability and agreement than the reviewers’ scores of long proposals in the old one-stage procedure. Our findings suggest that a two-stage procedure can substantially improve the efficiency of grant writing and review, without harming the reliability of the evaluation.Keywords
This publication has 66 references indexed in Scilit:
- How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer reviewScientometrics, 2024
- Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluationJournal of the Association for Information Science and Technology, 2022
- Who gets Horizon 2020 research grants? Propensity to apply and probability to succeed in a two-step analysisScientometrics, 2016
- Validating Grant-Making Processes: Construct Validity of the 2013 Senior Corps RSVP Grant ReviewVOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 2016
- Time of Day and the Decision to Prescribe AntibioticsJAMA Internal Medicine, 2014
- Assessing the scientific and technological output of EU Framework Programmes: evidence from the FP6 projects in the ICT fieldScientometrics, 2011
- Rejecting and resisting Nobel class discoveries: accounts by Nobel LaureatesScientometrics, 2009
- Persistent nepotism in peer-reviewScientometrics, 2007
- Exploring size and agglomeration effects on public research productivityScientometrics, 2005
- Creative DisagreementThe Sciences, 1979