Identifying Causes of Test Unfairness: Manipulability and Separability
Preprint
Abstract
Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English vocabulary unfamiliarity and classroom learning barriers, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Conditional Separable EffectsJournal of the American Statistical Association, 2022
- A generalized theory of separable effects in competing event settingsLifetime Data Analysis, 2021
- Debiased machine learning of conditional average treatment effects and other causal functionsThe Econometrics Journal, 2020
- Separable Effects for Causal Inference in the Presence of Competing EventsJournal of the American Statistical Association, 2020
- Estimation and Inference of Heterogeneous Treatment Effects using Random ForestsJournal of the American Statistical Association, 2018
- Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is GoingLanguage Assessment Quarterly, 2007
- Making Things Happen2004
- CAUSATION AND RACEETS Research Report Series, 2003
- Causal Inference without CounterfactualsJournal of the American Statistical Association, 2000
- A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effectMathematical Modelling, 1986