Monitoring of Scoring Using the e‐rater® Automated Scoring System and Human Raters on a Writing Test
Authors:
Open Access
- 29 April 2014
- journal article
- research article
- Published by Wiley in ETS Research Report Series
- Vol. 2014 (1) , 1-21
Abstract
This article proposes and investigates several methodologies for monitoring the quality of constructed‐response (CR) scoring, both human and automated. There is an increased interest in the operational scoring of essays using both automated scoring and human raters. There is also evidence of rater effects—scoring severity and score inconsistency by human raters. Recently, automated scoring of CRs was successfully implemented with human scoring for operational programs (TOEFL® and GRE® tests); however, there is much that is not yet known about the performance of automated scoring systems. Hence, for quality assurance purposes, there is the need to provide a consistent and standardized approach to monitor the quality of the CR scoring over time and across programs. Monitoring the scoring results will help provide scores that are both fair and accurate for test takers and test users, enabling testing programs to detect and correct changes in the severity of scoring.Keywords
This publication has 16 references indexed in Scilit:
- EVALUATION OF THE E‐RATER® SCORING ENGINE FOR THE GRE® ISSUE AND ARGUMENT PROMPTSETS Research Report Series, 2012
- A Framework for Evaluation and Use of Automated ScoringEducational Measurement: Issues and Practice, 2012
- USE OF E‐RATER® IN SCORING OF THE TOEFL IBT® WRITING TESTETS Research Report Series, 2011
- A validity-based approach to quality control and assurance of automated scoringAssessment in Education: Principles, Policy & Practice, 2011
- Statistical Process Control Charts for Measuring and Monitoring Temporal Consistency of RatingsJournal of Educational Measurement, 2010
- Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category UseJournal of Educational Measurement, 2009
- The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment DataJournal of Educational and Behavioral Statistics, 2002
- The Rater Bundle ModelJournal of Educational and Behavioral Statistics, 2001
- Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch ModelJournal of Educational Measurement, 1994
- Quality Control Charts1Bell System Technical Journal, 1926