Sven E. Hug
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Sven E. Hug has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Lutz Bornmann
Administrative Headquarters of the Max Planck Society
6 shared publications - Michael Ochsner
FORS
6 shared publications Hans-Dieter Daniel
ETH Zurich
5 shared publications- Robin Haunschild
Max Planck Institute for Solid State Research
4 shared publications - Martin Brändle
University of Zürich
3 shared publications Michael Ochsner
FORS Swiss Centre of Expertise in the Social Sciences, University of Lausanne, Bâtiment Géopolis, 1015 Lausanne, Switzerland
2 shared publications
Latest Publications
- Open Access
- 25 January 2024
- Published by Springer Nature in Scientometrics
- Vol. 129 (3) , 1231-1253
How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review
Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees’ overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most—but not all—of the referees’ judgments. Future studies should therefore examine how referees’ judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.
- 25 January 2024
- Published by Springer Nature in Scientometrics
- Vol. 129 (3) , 1231-1253
Open AccessHow do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review
Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees’ overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most—but not all—of the referees’ judgments. Future studies should therefore examine how referees’ judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.
- Open Access
How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review
Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees' overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most - but not all - of the referees' judgments. Future studies should therefore examine how referees' judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.
Open AccessHow do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review
Little is known whether peer reviewers use the same evaluation criteria and how they integrate the criteria into their overall judgment. This study therefore proposed two assessment styles based on theoretical perspectives and normative positions. According to the case-by-case style, referees use many and different criteria, weight criteria on a case-by-case basis, and integrate criteria in a complex, non-mechanical way into their overall judgment. According to the uniform style, referees use a small fraction of the available criteria, apply the same criteria, weight the criteria in the same way, and integrate the criteria based on simple rules (i.e., fast-and-frugal heuristics). These two styles were examined using a unique dataset from a career funding scheme that contained a comparatively large number of evaluation criteria. A heuristic (fast-and-frugal trees) and a complex procedure (logistic regression) were employed to describe how referees integrate the criteria into their overall judgment. The logistic regression predicted the referees' overall assessment with high accuracy and slightly more accurately than the fast-and-frugal trees. Overall, the results of this study support the uniform style but also indicate that the uniform style needs to be revised as follows: referees use many criteria and integrate the criteria using complex rules. However, and most importantly, the revised style could describe most - but not all - of the referees' judgments. Future studies should therefore examine how referees' judgments can be characterized in those cases where the uniform style failed. Moreover, the evaluation process of referees should be studied in more empirical and theoretical detail.
- Open Access
- 1 November 2022
- Published by MIT Press in Quantitative Science Studies
- Vol. 3 (3) , 815-831
Towards theorizing peer review
Academic peer review is seriously undertheorized because peer review studies focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe the main characteristics of the peer review literature, which focuses mainly on journal and grant peer review. Based on these characteristics, I then argue why theory is useful in research on peer review, and I present some theoretical efforts on peer review. I conclude by encouraging peer review researchers to be more theoretically engaged and outline activities that theoretical work on peer review could involve. This invitation to theory-building complements recent roadmaps and calls that have emphasized that we need to have better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.
- 1 November 2022
- Published by MIT Press in Quantitative Science Studies
- Vol. 3 (3) , 815-831
Open AccessTowards theorizing peer review
Academic peer review is seriously undertheorized because peer review studies focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe the main characteristics of the peer review literature, which focuses mainly on journal and grant peer review. Based on these characteristics, I then argue why theory is useful in research on peer review, and I present some theoretical efforts on peer review. I conclude by encouraging peer review researchers to be more theoretically engaged and outline activities that theoretical work on peer review could involve. This invitation to theory-building complements recent roadmaps and calls that have emphasized that we need to have better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.
- 2 December 2021
- Published by Oxford University Press (OUP) in Research Evaluation
- Vol. 31 (1) , 104-117
Do peers share the same criteria for assessing grant applications?
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (10 criteria related to knowledge gaps; feasibility; rigour; comprehensibility and argumentation; academic relevance; competence and experience of the applicant) and a broad consensus that include the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increase from the consensus far to the consensus close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.
- 2 December 2021
- Published by Oxford University Press (OUP) in Research Evaluation
- Vol. 31 (1) , 104-117
Do peers share the same criteria for assessing grant applications?
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (10 criteria related to knowledge gaps; feasibility; rigour; comprehensibility and argumentation; academic relevance; competence and experience of the applicant) and a broad consensus that include the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increase from the consensus far to the consensus close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.
- Open Access
- 29 October 2021
- Published by Center for Open Science in MetaArXiv
Towards Theorizing Peer Review
Despite more than 50 years of research, academic peer review and its contexts remain seriously undertheorized. Studies on peer review focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe four main characteristics of the peer review literature (i.e., focus on phenomena; focus on the meritocratic legitimacy of peer review; focus on application and improvement; fragmentation). Based on these characteristics, I then argue why theory is useful in research on peer review, and I present some theoretical efforts on peer review. I conclude by encouraging peer review researchers to be more theoretically engaged and outline activities that theoretical work on peer review could involve (i.e., defining peer review; determining main theoretical questions; reviewing and synthesizing theoretical contributions; identifying phenomena; building and modifying theories; reflecting on theorizing and theory). This invitation to theory building complements recent roadmaps and calls that have emphasized that we need to have better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.
- 29 October 2021
- Published by Center for Open Science in MetaArXiv
Open AccessTowards Theorizing Peer Review
Despite more than 50 years of research, academic peer review and its contexts remain seriously undertheorized. Studies on peer review focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe four main characteristics of the peer review literature (i.e., focus on phenomena; focus on the meritocratic legitimacy of peer review; focus on application and improvement; fragmentation). Based on these characteristics, I then argue why theory is useful in research on peer review, and I present some theoretical efforts on peer review. I conclude by encouraging peer review researchers to be more theoretically engaged and outline activities that theoretical work on peer review could involve (i.e., defining peer review; determining main theoretical questions; reviewing and synthesizing theoretical contributions; identifying phenomena; building and modifying theories; reflecting on theorizing and theory). This invitation to theory building complements recent roadmaps and calls that have emphasized that we need to have better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.
- Open Access
Do peers share the same criteria for assessing grant applications?
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (ten criteria related to knowledge gaps, feasibility, rigour, comprehensibility and argumentation, and academic relevance, as well as to the competence and experience of the applicant) and a broad consensus that includes the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increases from the consensus-far to the consensus-close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.
Open AccessDo peers share the same criteria for assessing grant applications?
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (ten criteria related to knowledge gaps, feasibility, rigour, comprehensibility and argumentation, and academic relevance, as well as to the competence and experience of the applicant) and a broad consensus that includes the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increases from the consensus-far to the consensus-close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.
- 22 July 2020
- Published by Ediciones Profesionales de la Informacion SL in Profesional de la información
- Vol. 29 (4)
Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note
When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research are being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publication and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Recently, however Bornmann and Marewski (2019) proposed such a framework. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach (Gigerenzer; Todd; ABC Research Group, 1999) to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.
- 22 July 2020
- Published by Ediciones Profesionales de la Informacion SL in Profesional de la información
- Vol. 29 (4)
Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note
When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research are being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publication and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Recently, however Bornmann and Marewski (2019) proposed such a framework. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach (Gigerenzer; Todd; ABC Research Group, 1999) to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.
- Open Access
Bibliometrics-based heuristics: What is their definition and how can they be studied?
When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research is being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publications and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Yet, Bornmann and Marewski [1] proposed such a framework recently. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach [2] to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.
Open AccessBibliometrics-based heuristics: What is their definition and how can they be studied?
When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research is being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publications and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Yet, Bornmann and Marewski [1] proposed such a framework recently. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach [2] to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.
- Open Access
- 10 March 2020
- Published by Springer Nature in Palgrave Communications
- Vol. 6 (1) , 1-15
Criteria for assessing grant applications: a systematic review
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
- 10 March 2020
- Published by Springer Nature in Palgrave Communications
- Vol. 6 (1) , 1-15
Open AccessCriteria for assessing grant applications: a systematic review
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
- Open Access
- 6 August 2019
- Published by Center for Open Science in PsyArXiv
Criteria for assessing grant applications: A systematic review
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e. the object of evaluation) and an evaluation criterion (i.e. the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification) as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.
- 6 August 2019
- Published by Center for Open Science in PsyArXiv
Open AccessCriteria for assessing grant applications: A systematic review
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e. the object of evaluation) and an evaluation criterion (i.e. the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification) as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non-Western countries, and examine fields other than the medical and health sciences.