Toward Automated Multi-trait Scoring of Essays: Investigating Links among Holistic, Analytic, and Text Feature Scores
Authors:
- 25 November 2009
- journal article
- research article
- Published by Oxford University Press (OUP) in Applied Linguistics
- Vol. 31 (3) , 391-417
Abstract
The main purpose of the study was to investigate the distinctness and reliability of analytic (or multi-trait) rating dimensions and their relationships to holistic scores and e-rater® essay feature variables in the context of the TOEFL® computer-based test (TOEFL CBT) writing assessment. Data analyzed in the study were holistic and multi-trait essay scores provided by human raters and essay feature variable scores computed by e-rater® (version 2.0) for two TOEFL CBT writing prompts. It was found that (i) all of the six multi-trait scores were not only correlated among themselves but also correlated with the holistic score, (ii) high correlations obtained among holistic and multi-trait scores were largely attributable to the impact of essay length on both holistic and multi-trait scoring, and (iii) some strong associations were confirmed between several e-rater variables and multi-trait rating dimensions. Implications are discussed for improving the multi-trait scoring of essays, refining e-rater essay feature variables, and validating automated essay scores.Keywords
This publication has 28 references indexed in Scilit:
- Detecting errors in English article usage by non-native speakersNatural Language Engineering, 2006
- Decision Making while Rating ESL/EFL Writing Tasks: A Descriptive FrameworkThe Modern Language Journal, 2002
- Using Generalizability Theory to Estimate the Reliability of Writing Scores Derived from Holistic and Analytical Scoring MethodsEducational and Psychological Measurement, 1999
- A vocabulary-size test of controlled productive abilityLanguage Testing, 1999
- Rating Nonnative Writing: The Trouble with Holistic ScoringTESOL Quarterly, 1995
- Looking behind the Curtain: What Do L2 Composition Ratings Really Mean?TESOL Quarterly, 1995
- Patterns of Writing Skills Derived From Cluster Analysis of Direct-Writing AssessmentsApplied Measurement in Education, 1994
- Lexical and Syntactic Features of ESL Writing by Students at Different Levels of L2 ProficiencyTESOL Quarterly, 1994
- The Literature of Direct Writing Assessment: Major Concerns and Prevailing TrendsReview of Educational Research, 1990
- Professors' Reactions to the Academic Writing of Nonnative-Speaking StudentsTESOL Quarterly, 1988