Evaluating Prototype Tasks and Alternative Rating Schemes for a New ESL Writing Test through G-theory
Authors:
- 14 November 2007
- journal article
- research article
- Published by Taylor & Francis in International Journal of Testing
- Vol. 7 (4) , 353-385
Abstract
Possible integrated and independent tasks were pilot tested for the writing section of a new generation of the TOEFL® (Test of English as a Foreign Language™). This study examines the impact of various rating designs and of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the perspective of generalizability theory (G-theory). Both univariate and multivariate G-theory analyses were conducted. It was found that (a) in terms of maximizing the score dependability, it would be more efficient to increase the number of tasks rather than the number of raters per essay; (b) two particular single-rating designs of “having different tasks for the same examinee rated by different raters” [p × (R:T), R:(p × T)] achieved relatively higher score dependability than other single-rating designs; and (c) a somewhat larger gain in composite score reliability was achieved when the number of listening–writing tasks was larger than that of reading–writing tasks.Keywords
This publication has 26 references indexed in Scilit:
- Generalizability Theory2001
- Performance Assessments from the Perspective of Generalizability TheoryApplied Psychological Measurement, 2000
- Accounting for nonsystematic error in performance ratingsLanguage Testing, 1996
- Generalizability of Performance AssessmentsEducational Measurement: Issues and Practice, 1995
- Investigating variability in tasks and rater judgements in a performance test of foreign language speakingLanguage Testing, 1995
- Generalizability Analyses of Work Keys Listening and Writing TestsEducational and Psychological Measurement, 1995
- Generalizability of Large-Scale Performance Assessments in Science: Promises and ProblemsApplied Measurement in Education, 1994
- Quality Control in the Development and Use of Performance AssessmentsApplied Measurement in Education, 1991
- Expertise in evaluating second language compositionsLanguage Testing, 1990
- AN INTRODUCTION TO GENERALIZABILITY THEORY IN SECOND LANGUAGE RESEARCH1Language Learning, 1982