The impact of differential item functioning on ability estimation using the Korean Medical License Examination with computerized adaptive testing: a post-hoc simulation study
Open Access
- 10 October 2025
- journal article
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 22, 31
Abstract
Purpose: This study examined the impact of differential item functioning (DIF) on ability estimation in a computerized adaptive testing (CAT) environment using real response data from the 2017 Korean Medical Licensing Examination (KMLE). We hypothesized that excluding gender-based DIF items would improve estimation accuracy, particularly for examinees at the extremes of the ability scale.Methods: The study was conducted in 2 steps: (1) DIF detection and (2) post-hoc simulation. The analysis used data from 3,259 examinees who completed all 360 dichotomous items. Gender-based DIF was detected with the residual-based DIF method (reference group: males; focal group: females). Two CAT conditions (all items vs. DIF-excluded) were compared against a “true θ” estimated from a fixed-form test of 264 non-DIF items. Accuracy was evaluated using bias, root mean square error (RMSE), and correlation with true θ.Results: In the CAT condition excluding DIF items, accuracy improved, with RMSE reduced and correlation with true θ increased. However, bias was slightly larger in magnitude. Gender-specific analyses showed that DIF removal reduced the underestimation of female ability but increased the underestimation of male ability, yielding estimates that were fairer across genders. When DIF items were included, estimation errors were more pronounced at both low and high ability levels.Conclusion: Managing DIF in CAT-based high-stakes examinations can enhance fairness and precision. Using real examinee data, this study provides practical evidence of the implications of DIF for CAT-based measurement and supports fairness-oriented test design.Keywords
Funding Information
- Hallym University (HRF-202504-006)
This publication has 11 references indexed in Scilit:
- Effect of Differential Item Functioning on Computer Adaptive Testing Under Different ConditionsApplied Psychological Measurement, 2024
- Detecting Differential Item Functioning in CAT Using IRT Residual DIF ApproachJournal of Educational Measurement, 2023
- Impacts of DIF on Ability Estimation in CATKorean Society for Educational Evaluation, 2022
- A Residual‐Based Differential Item Functioning Detection Framework in Item Response TheoryJournal of Educational Measurement, 2022
- Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing ExaminationJournal of Educational Evaluation for Health Professions, 2018
- Random Generation of Response Patterns under Computerized Adaptive Testing with theRPackagecatRJournal of Statistical Software, 2012
- Evaluation of the CATSIB DIF Procedure in a Pretest SettingJournal of Educational and Behavioral Statistics, 2004
- Adaptive EAP Estimation of Ability in a Microcomputer EnvironmentApplied Psychological Measurement, 1982
- Probabilistic Models for some Intelligence and Attainment Tests.Journal of the American Statistical Association, 1982
- A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental TestingJournal of the American Statistical Association, 1975