CONSTRUCT VALIDITY OF E‐RATER® IN SCORING TOEFL® ESSAYS
Author:
Open Access
- 1 June 2007
- journal article
- research article
- Published by Wiley in ETS Research Report Series
- Vol. 2007 (1)
Abstract
This study examined the construct validity of the e‐rater® automated essay scoring engine as an alternative to human scoring in the context of TOEFL® essay writing. Analyses were based on a sample of students who repeated the TOEFL within a short time period. Two e‐rater scores were investigated in this study, the first based on optimally predicting the human essay score and the second based on equal weights for the different features of e‐rater.Within a multitrait‐multimethod approach, the correlations and reliabilities of human and e‐rater scores were analyzed together with TOEFL subscores (structured writing, reading, and listening) and with essay length. Possible biases between human and e‐rater scores were examined with respect to differences in performance across countries of origin and differences in difficulty across prompts. Finally, a factor analysis was conducted on the e‐rater features to investigate the interpretability of their internal structure and determine which of the two e‐rater scores reflects this structure more closely.Results showed that the e‐rater score based on optimally predicting the human score measures essentially the same construct as human‐based essay scores with significantly higher reliability and consequently higher correlations with related language scores. The equal‐weights e‐rater score showed the same high reliability but significantly lower correlation with essay length. It is also aligned with the 3‐factor hierarchical (word use, grammar, and discourse) structure that was discovered in the factor analysis. Both e‐rater scores also successfully replicate human score differences between countries and prompts.Keywords
This publication has 11 references indexed in Scilit:
- A Review of Strategies for Validating Computer-Automated ScoringApplied Measurement in Education, 2002
- Comparing the Validity of Automated and Human Scoring of EssaysJournal of Educational Computing Research, 2002
- Stumping e-rater:challenging the validity of automated essay scoringComputers in Human Behavior, 2002
- Validity and Automad Scoring: It's Not Only the ScoringEducational Measurement: Issues and Practice, 1998
- COMPUTER ANALYSIS OF ESSAY CONTENT FOR AUTOMATED SCORE PREDICTION: A PROTOTYPE AUTOMATED SCORING SYSTEM FOR GMAT ANALYTICAL WRITING ASSESSMENT ESSAYSETS Research Report Series, 1998
- An introduction to latent semantic analysisDiscourse Processes, 1998
- FACTORS IN PERFORMANCE ON BRIEF, IMPROMPTU ESSAY EXAMINATIONSETS Research Report Series, 1995
- Computer Grading of Student Prose, Using Modern Concepts and SoftwareThe Journal of Experimental Education, 1994
- Construct validity: Construct representation versus nomothetic span.Psychological Bulletin, 1983
- Estimating coefficients in linear models: It don't make no nevermind.Psychological Bulletin, 1976