Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices
Open Access
- 28 September 2012
- journal article
- review article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 7 (9) , e46054
Abstract
Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria. We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest. Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.Keywords
This publication has 37 references indexed in Scilit:
- Non-Financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in FrancePLOS ONE, 2012
- Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysisBMJ, 2011
- Surveys of current status in biomedical science grant review: funding organisations' and grant reviewers' perspectivesBMC Medicine, 2010
- The Delphi process: a solution for reviewing novel grant applicationsInternational Journal of General Medicine, 2010
- Funding the UnfundablePharmacoEconomics, 2010
- Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization ProgrammesPLOS ONE, 2008
- Training of Peer Reviewers: Validation of a 5-Point Rating ScalePLOS Medicine, 2007
- The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review QualityPLOS Medicine, 2007
- Why do peer reviewers decline to review? A surveyJournal of Epidemiology and Community Health, 2006
- Quality Review in the Peer Review Organization ProgramQuality Assurance and Utilization Review, 1987