A comparison of quasi-experimental methods with data before and after an intervention: an introduction for epidemiologists and a simulation study
Authors:
- 6 April 2023
- journal article
- research article
- Published by Oxford University Press (OUP) in International Journal of Epidemiology
- Vol. 52 (5) , 1522-1533
Abstract
Background: As the interest in and use of quasi-experimental methods to evaluate impacts of health policies have dramatically increased in the epidemiological literature, we set out this study to (i) systematically compare several quasi-experimental methods that use data before and after an intervention and contrast their performance within a simulation framework while providing a brief overview of the methods; and (ii) discuss challenges that could arise from using these methods as well as directions for future research in the context of epidemiological applications. Methods: We considered single-group designs [pre-post and interrupted time series (ITS)] and multiple-group designs [controlled interrupted time series/difference-in-differences, synthetic control methods (SCMs): traditional SCMs and generalized SCMs]. We assessed performance based on bias and root mean squared error. Results: We identified settings in which each method failed to provide unbiased estimates. We found that, among the methods investigated, when data for multiple time points and for multiple control groups are available (multiple-group designs), data-adaptive methods such as the generalized SCM were generally less biased than other methods evaluated in our study. In addition, when all of the included units have been exposed to treatment (single-group designs) and data for a sufficiently long pre-intervention period are available, then the ITS performs very well, provided the underlying model is correctly specified. Conclusions: When using a quasi-experimental method using data before and after an intervention, epidemiologists should strive to use, whenever feasible, data-adaptive methods that nest alternative identifying assumptions including relaxing the parallel trend assumption (e.g. generalized SCMs).Keywords
Funding Information
- National Institute on Minority Health and Health Disparities (1K01MD014163-01A1)
- National Institutes of Health
This publication has 27 references indexed in Scilit:
- Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention ResearchAnnual Review of Public Health, 2017
- Evaluating the Health Impact of Large-Scale Public Policy Changes: Classical and Novel ApproachesAnnual Review of Public Health, 2017
- Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects ModelsPolitical Analysis, 2017
- Estimating causal effects: considering three alternatives to difference-in-differences estimationHealth Services and Outcomes Research Methodology, 2016
- Interrupted time series analysis in drug utilization research is increasing: systematic review and recommendationsJournal of Clinical Epidemiology, 2015
- Using propensity scores in difference-in-differences models to estimate the effects of a policy changeHealth Services and Outcomes Research Methodology, 2014
- Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control ProgramJournal of the American Statistical Association, 2010
- Participation in Cancer Clinical TrialsJAMA, 2004
- Statistics and Causal InferenceJournal of the American Statistical Association, 1986
- Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of Educational Psychology, 1974