Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination
Authors:
- 15 June 2001
- journal article
- Published by Wiley in Journal of Educational Measurement
- Vol. 38 (2) , 121-145
Abstract
In this study, patterns of variation in severities of a group of raters over time or so‐called “rater drift” was examined when raters scored an essay written under examination conditions. At the same time feedback was given to rater leaders (called “table leaders”) who then interpreted the feedback and reported to the raters. Rater severities in five successive periods were estimated using a modified linear logistic test model (LLTM, Fischer, 1973) approach. It was found that the raters did indeed drift towards the mean, but a planned comparision of the feedback with a control condition was not successful; it was believed that this was due to contamination at the table leader level. A series of models was also estimated designed to detect other types of rater effects beyond severity: a tendency to use extreme scores, and tendency to prefer certain categories. The models for these effects were found to be showing significant improvement in fit, implying that these effects were indeed present, although they were difficult to detect in relatively short time periods.Keywords
This publication has 19 references indexed in Scilit:
- Evaluating Rater Accuracy in Performance AssessmentsJournal of Educational Measurement, 1996
- Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch ModelJournal of Educational Measurement, 1994
- Sampling Variability of Performance AssessmentsJournal of Educational Measurement, 1993
- Least Squares Models to Correct for Rater Effects in Performance AssessmentJournal of Educational Measurement, 1993
- Measuring Attitudes With a Threshold Model Drawing on a Traditional Scaling ConceptApplied Psychological Measurement, 1988
- The Analysis of Partial Credit ScoringApplied Measurement in Education, 1988
- Understanding Scoring Reliability: Experiments in Calibrating Essay ReadersJournal of Educational Statistics, 1988
- Understanding Scoring Reliability: Experiments in Calibrating Essay ReadersJournal of Educational Statistics, 1988
- Rating the ratings: Assessing the psychometric quality of rating data.Psychological Bulletin, 1980
- Reliability of repeated grading of essay type examinations.Journal of Educational Psychology, 1930