Individual Feedback to Enhance Rater Training: Does It Work?
Authors:
- Cathie ElderCECathie Elder
- Ute KnochUKUte Knoch
- Gary BarkhuizenGBGary Barkhuizen
- Janet von RandowJRJanet von Randow
- 1 October 2005
- journal article
- Published by Taylor & Francis in Language Assessment Quarterly
- Vol. 2 (3) , 175-196
Abstract
Research on the utility of feedback to raters in the form of performance reports has produced mixed findings (Lunt, Morton, & Wigglesworth, 1994; Wigglesworth, 1993) and has thus far been trialled only in oral assessment contexts. This article reports on a study investigating raters' attitudes and responsiveness to feedback on their ratings of an analytically scored writing task administered in the context of a large-scale university writing assessment program. After participating in an online refresher training program, 50 scripts were rated independently by 8 raters. A multifaceted Rasch analysis (using FACETS; Linacre & Wright, 1993) was used to generate individualized reports on each rater's pattern of bias with respect to particular categories of the rating scale as well as their overall consistency and relative severity in relation to others in the group. Reports were explicated at a group briefing session, and raters were asked to attend to the feedback when scoring a further batch of 60 scripts. Q...Keywords
This publication has 11 references indexed in Scilit:
- Assessment criteria in a large-scale writing test: what do they really mean to the raters?Language Testing, 2002
- Teachers' perceptions of on-line rater training and monitoringSystem, 2001
- Systematic effects in the rating of second-language speaking ability: test method and learner discourseLanguage Testing, 1999
- Scaling descriptors for language proficiency scalesLanguage Testing, 1998
- Using FACETS to model rater training effectsLanguage Testing, 1998
- The development of a common framework scale of descriptors of language proficiency based on a theory of measurementSystem, 1995
- Constructing rating scales for second language testsELT Journal, 1995
- Effects of training on raters of ESL compositionsLanguage Testing, 1994
- Reliability, Validity, and Holistic Scoring: What We Know and What We Need to KnowCollege Composition and Communication, 1990
- A Deterministic Theory of Clinical Performance RatingEvaluation & the Health Professions, 1984