Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments
- 12 July 2019
- journal article
- research article
- Published by Wiley in Educational Measurement: Issues and Practice
- Vol. 38 (3) , 97-111
Abstract
For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.Keywords
This publication has 37 references indexed in Scilit:
- Modeling Common Traits and Method Effects in Multitrait-Multimethod AnalysisMultivariate Behavioral Research, 2010
- Computer- vs. paper-based tasks: Are they equivalent?Ergonomics, 2008
- Computer-Based and Paper-and-Pencil Administration Mode Effects on a Statewide End-of-Course English TestEducational and Psychological Measurement, 2008
- Modelling Method Effects as Individual Causal EffectsJournal of the Royal Statistical Society Series A: Statistics in Society, 2007
- Comparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading AssessmentsEducational and Psychological Measurement, 2007
- The Mode Effect: A Literature Review of Human and Technological Issues in Computerized TestingInternational Journal of Testing, 2006
- Speed and Performance Differences among Computer-Based and Paper-Pencil TestsJournal of Educational Computing Research, 2004
- Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis.Psychological Bulletin, 1993
- TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTSJournal of Educational Measurement, 1984
- Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin, 1959