Retrofitting Non-diagnostic Reading Comprehension Assessment: Application of the G-DINA Model to a High Stakes Reading Comprehension Test
- 27 May 2019
- journal article
- research article
- Published by Taylor & Francis in Language Assessment Quarterly
- Vol. 16 (3) , 294-311
Abstract
With the advancement of Cognitive Diagnostic Assessment (CDA) and the pertinent statistical models, different domains of large-scale testing and assessment have been examined for the sake of reporting more diagnostic information. Applying the generalized deterministic input, noisy, “and” gate (G-DINA) model, the current study analyzed a high-stakes L2 reading comprehension test as an integral section of the PhD degree’s entrance exam in Iran. It aimed at examining the reading comprehension attributes underlying this high stakes test in an attempt to check the capability of CDA models, (in this case G-DINA), in providing diagnostic information for test developers and users. The items’ response data were analyzed in R, “GDINA” package, version 1.4.2. With data collected from multiple sources including the current literature on the sub-skills of reading comprehension, test specifications, test-takers’ think-aloud verbal protocols, and expert panel’s judgments, an initial Q-matrix, including five sub-skills, was developed and then validated. Data analysis, using the validated version of Q-matrix, showed the sub-skill (attribute) prevalence and its difficulty. Code-related sub-skills were the easiest and the most prevalent ones and the connecting/synthesizing sub-skills were the most difficult and the least prevalent ones for the test-takers. Skill mastery profile results verified the relationship among the sub-skills of reading skill and also the prominence of some of these sub-skills over others.Keywords
This publication has 37 references indexed in Scilit:
- DINA Model and Parameter Estimation: A DidacticJournal of Educational and Behavioral Statistics, 2009
- Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessmentLanguage Testing, 2009
- Why Cognitive Diagnostic Assessment?2007
- The Fusion Model Skills Diagnosis System2007
- Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response TheoryApplied Psychological Measurement, 2001
- Test impact revisited: washback effect over timeLanguage Testing, 1996
- An exploratory study into the construct validity of a reading comprehension test: triangulation of data sourcesLanguage Testing, 1991
- A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification ApproachJournal of Educational Statistics, 1985
- Estimating the Dimension of a ModelThe Annals of Statistics, 1978
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974