Detecting Item Preknowledge Using a Predictive Checking Method
Authors:
- 22 January 2017
- journal article
- research article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 41 (4) , 243-263
Abstract
Repeatedly using items in high-stake testing programs provides a chance for test takers to have knowledge of particular items in advance of test administrations. A predictive checking method is proposed to detect whether a person uses preknowledge on repeatedly used items (i.e., possibly compromised items) by using information from secure items that have zero or very low exposure rates. Responses on the secure items are first used to estimate a person’s proficiency distribution, and then the corresponding predictive distribution for the person’s responses on the possibly compromised items is constructed. The use of preknowledge is identified by comparing the observed responses to the predictive distribution. Different estimation methods for obtaining a person’s proficiency distribution and different choices of test statistic in predictive checking are considered. A simulation study was conducted to evaluate the empirical Type I error and power rate of the proposed method. The simulation results suggested that the Type I error of this method is well controlled, and this method is effective in detecting preknowledge when a large proportion of items are compromised even with a short secure section. An empirical example is also presented to demonstrate its practical use.Keywords
Funding Information
- Educational Testing Service (Harold Gulliksen Psychometric Research Fellowship)
This publication has 21 references indexed in Scilit:
- Generalized Fiducial Inference: A Review and New ResultsJournal of the American Statistical Association, 2016
- Detecting Item Preknowledge in Computerized Adaptive Testing Using Information Theory and Combinatorial OptimizationJournal of Computerized Adaptive Testing, 2014
- Bayesian Data Analysis2013
- Detection of Test Collusion via Kullback–Leibler DivergenceJournal of Educational Measurement, 2013
- Improving Person‐Fit Assessment by Correcting the Ability Estimate and Its Reference DistributionJournal of Educational Measurement, 2008
- A Bayesian Approach to Person Fit Analysis in Item Response Theory ModelsApplied Psychological Measurement, 2003
- Interval Estimation for a Binomial ProportionStatistical Science, 2001
- Detecting Inappropriate Test Scores with Optimal and Practical Appropriateness IndicesApplied Psychological Measurement, 1987
- Appropriateness measurement with polychotomous item response models and standardized indicesBritish Journal of Mathematical and Statistical Psychology, 1985
- Inverse ProbabilityMathematical Proceedings of the Cambridge Philosophical Society, 1930