Assessor experience, not rubric type, determines grading reliability in biosciences coursework
Open Access
- 22 April 2026
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Education
- Vol. 11, 1729644
Abstract
Introduction: Previous research on whether holistic or analytical scoring rubrics yield higher reliability and validity has been mixed and inconclusive, with little of it based on empirical data. This study addressed this gap by comparing scores from assessors with varying experience levels who used both a holistic and an analytical scale descriptor to grade undergraduate bioscience essays, complemented by a qualitative survey of assessor perceptions. The goal was to determine which method resulted in more consistent inter-rater agreement. Methods: For the study, independent assessors (two per essay) scored essays using either holistic ( n = 212) or analytical ( n = 62) scale descriptors, over four consecutive years. Each assessor provided both a holistic and an analytical score for every essay. The agreement between the scores was then calculated. Results: The results showed no significant difference in scores awarded holistically versus analytically for the same essay, regardless of the scale descriptor used or the assessor's experience. However, a key finding was that experienced assessors had a higher agreement with the final awarded grade than their less-experienced counterparts. Discussion: This suggests that the experience of the assessor, rather than the specific scoring rubric or guidance provided, is the primary factor in determining the reliability of the grade awarded.Keywords
This publication has 47 references indexed in Scilit:
- Rating writing: Comparison of holistic and analytic grading approaches in pre-service teachersLearning and Instruction, 2024
- Judgment accuracy of German student texts: Do teacher experience and content knowledge matter?Teaching and Teacher Education, 2022
- A critical review of the arguments against the use of rubricsEducational Research Review, 2020
- Best practice guidance for linear mixed-effects models in psychological scienceJournal of Memory and Language, 2020
- Developing and examining validity evidence for the Writing Rubric to Inform Teacher Educators (WRITE)Assessing Writing, 2019
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- Generalized linear mixed models: a practical guide for ecology and evolutionTrends in Ecology & Evolution, 2009
- The use of scoring rubrics: Reliability, validity and educational consequencesEducational Research Review, 2007
- Assessing Writing2002
- Writing evaluation: what can analytic versus holistic essay scoring tell us?System, 2001