Privacy-preserving impact evaluation using Difference-in-Differences
Authors:
- Manuel HuthMHManuel Huth
- Lea SeepLSLea Seep
- Francisco SaúteFSFrancisco Saúte
- Elisa SicuriESElisa Sicuri
- Jan HasenauerJHJan Hasenauer
Preprint
Abstract
Summary: Difference-in-Differences (DID) is a widely used tool for causal impact evaluation but is constrained by data privacy regulations when applied to sensitive personal information, such as individual-level performance records or healthcare data, that must not be shared with data analysts. Obtaining consent can reduce sample sizes or exclude treated/untreated groups, diminishing statistical power or making estimation impossible. Federated Learning, which shares aggregated statistics to ensure privacy, can address these concerns, but advanced federated DID software packages remain scarce. We derived and developed a federated version of the Callaway and Sant’Anna DID, implemented within the DataSHIELD platform. Our package adheres to DataSHIELD’s security measures and adds extra protections, enhancing data privacy and confidentiality. It reproduces point estimates, asymptotic standard errors, and bootstrapped standard errors equivalent to the non-federated implementation. We demonstrate this functionality on simulated data and real-world data from a malaria intervention in Mozambique. By leveraging federated estimates, we increase effective sample sizes leading to reduced estimation uncertainty, and enable estimation when single data owners cannot share the data but only have access to the treated or untreated group.Keywords
This publication has 25 references indexed in Scilit:
- Multi-site fMRI analysis using privacy-preserving federated learning and domain adaptation: ABIDE resultsMedical Image Analysis, 2020
- Using Difference-in-Differences to Identify Causal Effects of COVID-19 PoliciesSSRN Electronic Journal, 2020
- DataSHIELD – New Directions and DimensionsData Science Journal, 2017
- Health and Access to Care during the First 2 Years of the ACA Medicaid ExpansionsNew England Journal of Medicine, 2017
- Beverage purchases from stores in Mexico under the excise tax on sugar sweetened beverages: observational studyBMJ, 2016
- The effect of medical marijuana laws on adolescent and adult use of marijuana, alcohol, and other substancesJournal of Health Economics, 2015
- DataSHIELD: taking the analysis to the data, not the data to the analysisInternational Journal of Epidemiology, 2014
- Comparative Politics and the Synthetic Control MethodAmerican Journal of Political Science, 2014
- Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control ProgramJournal of the American Statistical Association, 2010
- Water for Life: The Impact of the Privatization of Water Services on Child MortalityJournal of Political Economy, 2005