Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation
- 1 February 2016
- journal article
- research article
- Published by Taylor & Francis in Educational Psychology
- Vol. 37 (1) , 8-25
Abstract
An increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggest growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses. A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level English as a Second Language (ESL) writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilisation inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students’ abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers’ approach to accuracy, and instructors’ assumptions about the underlying purposes of AWE-based writing activities.Keywords
This publication has 27 references indexed in Scilit:
- Formative assessment: a critical reviewAssessment in Education: Principles, Policy & Practice, 2011
- The promise of NLP and speech processing technologies in language assessmentLanguage Testing, 2010
- The utility of article and preposition error correction systems for English language learners: Feedback and assessmentLanguage Testing, 2010
- Toward a Taxonomy of Written Errors: Investigation Into the Written Errors of Hong Kong Cantonese ESL LearnersTESOL Quarterly, 2010
- Automated Grammatical Error Detection for Language LearnersSynthesis Lectures on Human Language Technologies, 2010
- Answering the Call for a Standard Reliability Measure for Coding DataCommunication Methods and Measures, 2007
- Detecting errors in English article usage by non-native speakersNatural Language Engineering, 2006
- Help Seeking and Help Design in Interactive Learning EnvironmentsReview of Educational Research, 2003
- An argument-based approach to validity.Psychological Bulletin, 1992
- A Cognitive Process Theory of WritingCollege Composition and Communication, 1981