Investigation of Interrater Reliability in The Evaluation of Foreign Language Writing Skills With Multigroup Confirmatory Factor Analysis
Authors:
- Emine ÖnenEÖEmine Önen
- Melike Kübra Taşdelen YayvakMYMelike Kübra Taşdelen Yayvak
Open Access
- 3 December 2018
- journal article
- Published by Redfame Publishing in Journal of Education and Training Studies
- Vol. 7 (1) , 30-37
Abstract
In this study, it was aimed to examine the interrater reliability of the scoring of paragraph writing skills on foreign languages with the measurement invariance tests. The study group consists of 267 students studying English at the Preparatory School at Gazi University. In the study, where students write a paragraph on the same topic, the paragraphs are rated separately by three different interrater using the same scoring key. The evidence for the validity measurements was collected with AFA and DFA while the evidence for the reliability measurements was collected by the Cronbach-alpha (α) coefficient. As a result of testing with Multi-Group Confirmatory Factor Analysis within the context of the measurement invariance of the interrater reliability, no evidence of full and partial scalar invariance can be obtained while evidence of formal configural and metric invariance is obtained. As a result, the lack of evidence of scalar invariance means that raters scoring the writing skills do not use the same initial level of performance. In this case, the invariant uniqueness and invariant factor variances could not be tested, and therefore no evidence of reliability between raters could be obtained.Keywords
This publication has 19 references indexed in Scilit:
- Reliability-Based Feature Weighting for Automated Essay ScoringApplied Psychological Measurement, 2014
- A checklist for testing measurement invarianceEuropean Journal of Developmental Psychology, 2012
- Self-, peer-, and teacher-assessments in Japanese university EFL writing classroomsLanguage Testing, 2009
- How accurate are ESL students’ holistic writing scores on large-scale assessments?—A generalizability theory approachAssessing Writing, 2008
- Rating scale impact on EFL essay marking: A mixed-method studyAssessing Writing, 2007
- Data Equivalence in Cross-Cultural Research: A Comparison of Classical Test Theory and Latent Trait Theory Based ApproachesAustralasian Marketing Journal, 1999
- Using FACETS to model rater training effectsLanguage Testing, 1998
- Rater characteristics and rater bias: implications for trainingLanguage Testing, 1995
- Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch ModelJournal of Educational Measurement, 1994
- Feedback in the writing process: a model and methods for implementationELT Journal, 1990