An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach
Author:
- 1 April 2009
- journal article
- research article
- Published by SAGE Publications in Language Testing
- Vol. 26 (2) , 187-217
Abstract
This study used a mixed methods research approach to examine how native English-speaking (NS) and non-native English-speaking (NNS) teachers assess students' oral English performance. The evaluation behaviors of two groups of teachers (12 Canadian NS teachers and 12 Korean NNS teachers) were compared with regard to internal consistency, severity, and evaluation criteria. Results of a Many-faceted Rasch Measurement analysis showed that most of the NS and NNS teachers maintained acceptable levels of internal consistency, with only one or two inconsistent raters in each group. The two groups of teachers also exhibited similar severity patterns across different tasks. However, substantial dissimilarities emerged in the evaluation criteria teachers used to assess students' performance. A qualitative analysis demonstrated that the judgments of the NS teachers were more detailed and elaborate than those of the NNS teachers in the areas of pronunciation, specific grammar use, and the accuracy of transferred information. These findings are used as the basis for a discussion of NS versus NNS teachers as language assessors on the one hand and the usefulness of mixed methods inquiries on the other.Keywords
This publication has 21 references indexed in Scilit:
- Rater judgment and English language speaking proficiencyWorld Englishes, 2005
- English as a Global Language2003
- Crafting mixed‐method evaluation designsNew Directions for Evaluation, 1997
- Deriving oral assessment scales across different tests and rater groupsLanguage Testing, 1995
- The effect of rater variables in the development of an occupation-specific language performance testLanguage Testing, 1995
- Data Analysis Strategies for Mixed-Method Evaluation DesignsEducational Evaluation and Policy Analysis, 1993
- 'Naive' native speakers and judgements of oral proficiency in SpanishLanguage Testing, 1989
- Toward a Conceptual Framework for Mixed-Method Evaluation DesignsEducational Evaluation and Policy Analysis, 1989
- Native and Nonnative Judgments of Intelligibility and IrritationLanguage Learning, 1987
- Perceptions of the Communicative Efforts of American Students of SpanishThe Modern Language Journal, 1980