Application of Think Aloud Protocols for Examining and Confirming Sources of Differential Item Functioning Identified by Expert Reviews
- 11 June 2010
- journal article
- Published by Wiley in Educational Measurement: Issues and Practice
- Vol. 29 (2) , 24-35
Abstract
This paper demonstrates and discusses the use of think aloud protocols (TAPs) as an approach for examining and confirming sources of differential item functioning(DIF). The TAPs are used to investigate to what extent surface characteristics of the items that are identified by expert reviews as sources of DIF are supported by empirical evidence from examinee thinking processes in the English and French versions of a Canadian national assessment. In this research, the TAPs confirmed sources of DIF identified by expert reviews for 10 out of 20 DIF items. The moderate agreement between TAPs and expert reviews indicates that evidence from expert reviews cannot be considered sufficient in deciding whether DIF items are biased and such judgments need to include evidence from examinee thinking processes.Keywords
This publication has 27 references indexed in Scilit:
- Language, Dialect, and Register: Sociolinguistics and the Estimation of Measurement Error in the Testing of English Language LearnersTeachers College Record: the Voice of Scholarship in Education, 2006
- Using Multiple-Variable Matching to Identify Cultural Sources of Differential Item FunctioningInternational Journal of Testing, 2006
- Assessment Accommodations for English Language Learners: Implications for Policy-Based Empirical ResearchReview of Educational Research, 2004
- Examining Language in Context: The Need for New Research and Practice Paradigms in the Testing of English-Language LearnersEducational Researcher, 2003
- Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory AnalysisJournal of Educational Measurement, 2001
- On the cultural validity of science assessmentsJournal of Research in Science Teaching, 2001
- Investigating the Cognitive Complexity of Science AssessmentsEducational Measurement: Issues and Practice, 1998
- Think‐aloud protocols and the study of comprehensionDiscourse Processes, 1996
- Generating Thinking-Aloud Protocols: Impact on the Narrative Writing of College StudentsThe American Journal of Psychology, 1995
- Analysis of Differential Item Functioning in the NAEP History AssessmentJournal of Educational Measurement, 1989