Evaluating rater responses to an online training program for L2 writing assessment
- 1 January 2007
- journal article
- research article
- Published by SAGE Publications in Language Testing
- Vol. 24 (1) , 37-64
Abstract
The use of online rater self-training is growing in popularity and has obvious practical benefits, facilitating access to training materials and rating samples and allowing raters to reorient themselves to the rating scale and self monitor their behaviour at their own convenience. However there has thus far been little research into rater attitudes to training via this modality and its effectiveness in enhancing levels of inter- and intra-rater agreement. The current study explores these issues in relation to an analytically-scored academic writing task designed to diagnose undergraduates’ English learning needs. 8 ESL raters scored a number of pre-rated benchmark writing samples online and received immediate feedback in the form of a discrepancy score indicating the gap between their own rating of the various categories of the rating scale and the official ratings assigned to the benchmark writing samples. A batch of writing samples was rated twice (before and after participating in the online training) by each rater and Multifaceted Rasch analyses were used to compare levels of rater agreement and rater bias (on each analytic rating category). Raters’ views regarding the effectiveness of the training were also canvassed. While findings revealed limited overall gains in reliability, there was considerable individual variation in receptiveness to the training input. The paper concludes with suggestions for refining the online training program and for further research into factors influencing rater responsiveness.Keywords
This publication has 13 references indexed in Scilit:
- Teachers' perceptions of on-line rater training and monitoringSystem, 2001
- The Stability of Rater Severity in Large‐Scale Assessment ProgramsJournal of Educational Measurement, 2000
- Using FACETS to model rater training effectsLanguage Testing, 1998
- Rater characteristics and rater bias: implications for trainingLanguage Testing, 1995
- Effects of training on raters of ESL compositionsLanguage Testing, 1994
- Judge Consistency and Severity Across Grading PeriodsEvaluation & the Health Professions, 1990
- Measuring the Impact of Judge Severity on Examination ScoresApplied Measurement in Education, 1990
- Reliability, Validity, and Holistic Scoring: What We Know and What We Need to KnowCollege Composition and Communication, 1990
- Researching Practice: Evaluating Assessment EssaysCollege Composition and Communication, 1986
- A Deterministic Theory of Clinical Performance RatingEvaluation & the Health Professions, 1984