The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability
Authors:
Open Access
- 23 May 2017
- journal article
- Published by Springer Nature in Palgrave Communications
- Vol. 3 (1)
Abstract
There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with an efficient review system, and the wider perception of a fair evaluation process. Therefore, there is an arguable role for a more structured approach to the peer review process during a time when evaluators are effectively novices in practice with the criterion, such as with societal impact. This article explores the consequences of a structured peer review process that aimed to increase inter-reviewer reliability within panels charged with assessing societal impact. Using a series of interviews from evaluators from the UK’s Research Excellence Framework conducted before (pre-evaluation) and then again after the completion of the process (post-evaluation), it explores evaluators’ perceptions about how one tool of a structured evaluation process, pre-evaluation training, influenced their approaches to achieving a consensus within the peer review panel. Building on lessons learnt from studies on achieving inter-reviewer reliability and from consensus building with peer review groups, this article debates the benefits of structured peer review processes in cases when the evaluators are unsure of the criterion (as was the case with the Impact criterion), and therefore the risks of a low IRR are increased. In particular, this article explores how individual approaches to assessing Impact were normalized during group deliberation around Impact and how these relate to evaluators’ perceptions of the advice given during the pre-evaluation training. This article is published as part of a collection on the future of research assessment.Keywords
This publication has 32 references indexed in Scilit:
- Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer PracticesPLOS ONE, 2012
- The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster?Research Evaluation, 2011
- Extraneous factors in judicial decisionsProceedings of the National Academy of Sciences, 2011
- Panel peer review of grant applications: what do we know from research in social psychology on judgment and decision-making in groups?Research Evaluation, 2010
- Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline GrantAccountability in Research, 2009
- Health research: measuring the social, health and economic benefitsCanadian Medical Association Journal, 2009
- Expert panels evaluating research: decision-making and sources of biasResearch Evaluation, 2004
- When nonreliability of reviews indicates solid scienceBehavioral and Brain Sciences, 1991
- Reliability, fairness, objectivity and other inappropriate goals in peer reviewBehavioral and Brain Sciences, 1991
- The Predictive Ability of Peer Review of Grant Proposals: The Case of Ecology and the US National Science FoundationSocial Studies of Science, 1991