Reliability-Based Feature Weighting for Automated Essay Scoring
Author:
- 2 December 2014
- journal article
- research article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 39 (4) , 303-313
Abstract
From their earliest origins, automated essay scoring systems strived to emulate human essay scores and viewed them as their ultimate validity criterion. Consequently, the importance (or weight) and even identity of computed essay features in the composite machine score were determined by statistical techniques that sought to optimally predict human scores from essay features. However, it is evident that machine evaluation of essays is fundamentally different from human evaluation and therefore is not likely to measure the same set of writing skills. As a consequence, feature weights of human-prediction machine scores (reflecting their importance in the composite scores) are bound to reflect statistical artifacts. This article suggests alternative feature weighting schemes based on the premise of maximizing reliability and internal consistency of the composite score. The article shows, in the context of a large-scale writing assessment, that these alternative weighting schemes are significantly different from human-prediction weights and give rise to comparable or even superior reliability and validity coefficients.Keywords
This publication has 12 references indexed in Scilit:
- A DIFFERENTIAL WORD USE MEASURE FOR CONTENT ANALYSIS IN AUTOMATED ESSAY SCORINGETS Research Report Series, 2011
- Rater types in writing performance assessments: A classification approach to rater variabilityLanguage Testing, 2008
- CONSTRUCT VALIDITY OF E‐RATER® IN SCORING TOEFL® ESSAYSETS Research Report Series, 2007
- Validity and Automad Scoring: It's Not Only the ScoringEducational Measurement: Issues and Practice, 1998
- Reliability of measurement in psychology: From Spearman-Brown to maximal reliability.Psychological Methods, 1996
- Rater characteristics and rater bias: implications for trainingLanguage Testing, 1995
- Construct validity: Construct representation versus nomothetic span.Psychological Bulletin, 1983
- Differential Weighting: A Review of Methods and Empirical StudiesReview of Educational Research, 1970
- A Unified Treatment of the Weighting ProblemPsychometrika, 1968
- Alpha-Maximized Factor Analysis (Alphamax): Its Relation to Alpha and Canonical Factor AnalysisPsychometrika, 1968