Diagnostic assessment of writing: A comparison of two rating scales
Author:
- 1 April 2009
- journal article
- research article
- Published by SAGE Publications in Language Testing
- Vol. 26 (2) , 275-304
Abstract
Alderson (2005) suggests that diagnostic tests should identify strengths and weaknesses in learners' use of language and focus on specific elements rather than global abilities. However, rating scales used in performance assessment have been repeatedly criticized for being imprecise and therefore often resulting in holistic marking by raters (Weigle, 2002). The aim of this study is to compare two rating scales for writing in an EAP context; one `a priori' developed scale with less specific descriptors of the kind commonly used in proficiency tests and one empirically developed scale with detailed level descriptors. The validation process involved 10 trained raters applying both sets of descriptors to the rating of 100 writing scripts yielded from a large-scale diagnostic assessment administered to both native and non-native speakers of English at a large university. A quantitative comparison of rater behaviour was undertaken using FACETS. Questionnaires and interviews were administered to elicit the raters' perceptions of the efficacy of the two types of scales. The results indicate that rater reliability was substantially higher and that raters were able to better distinguish between different aspects of writing when the more detailed descriptors were used. Rater feedback also showed a preference for the more detailed scale. The findings are discussed in terms of their implications for rater training and rating scale development.Keywords
This publication has 18 references indexed in Scilit:
- Exploring the Utility of a Web-Based English Language Screening ToolLanguage Assessment Quarterly, 2008
- Evaluating rater responses to an online training program for L2 writing assessmentLanguage Testing, 2007
- Re-training writing raters online: How does it compare with face-to-face training?Assessing Writing, 2007
- ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherenceAssessing Writing, 2007
- Assessment criteria in a large-scale writing test: what do they really mean to the raters?Language Testing, 2002
- 12. DISCOURSE AND ASSESSMENTAnnual Review of Applied Linguistics, 2002
- Scaling descriptors for language proficiency scalesLanguage Testing, 1998
- Does thick description lead to smart tests? A data-based approach to rating scale constructionLanguage Testing, 1996
- Principal Component Analysis1986
- The Application of Electronic Computers to Factor AnalysisEducational and Psychological Measurement, 1960