Missing, Presumed different: Quantifying the risk of Attrition Bias in Education Evaluations
Authors:
- 10 March 2021
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the Royal Statistical Society Series A: Statistics in Society
- Vol. 184 (2) , 732-760
Abstract
We estimate the magnitude of attrition bias for 10 randomized controlled trials (RCTs) in education. We make use of a unique feature of administrative school data in England that allows us to analyse post-test academic outcomes for nearly all students, including those who originally dropped out of the RCTs we analyse. We find that the typical magnitude of attrition bias is 0.015 effect size units (ES), with no estimate greater than 0.034 ES. This suggests that, in practice, the risk of attrition bias is limited. However, this risk should not be ignored as we find some evidence against the common ‘Missing At Random’ assumption. Attrition appears to be more problematic for treated units. We recommend that researchers incorporate uncertainty due to attrition bias, as well as performing sensitivity analyses based on the types of attrition mechanisms that are observed in practice.Keywords
This publication has 16 references indexed in Scilit:
- Differential attrition in health behaviour change trials: A systematic review and meta-analysisPsychology & Health, 2014
- Flaws in Evaluations of Social ProgramsEvaluation Review, 2014
- No differential attrition was found in randomized controlled trials published in general medical journals: a meta-analysisJournal of Clinical Epidemiology, 2013
- Fitting Multilevel Multivariate Models with Missing Data in Responses and Covariates that May Include Interactions and Non-Linear TermsJournal of the Royal Statistical Society Series A: Statistics in Society, 2013
- Multiple imputation for missing data in epidemiological and clinical research: potential and pitfallsBMJ, 2009
- A Re-Evaluation of Random-Effects Meta-AnalysisJournal of the Royal Statistical Society Series A: Statistics in Society, 2008
- What Is an Intracluster Correlation Coefficient? Crucial Concepts for Primary Care ResearchersThe Annals of Family Medicine, 2004
- Conditional Statistical Inference and Quantification of RelevanceJournal of the Royal Statistical Society Series B: Statistical Methodology, 2003
- A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes.Psychological Methods, 1998
- A sharper Bonferroni procedure for multiple tests of significanceBiometrika, 1988