Zero to Hero: Exploiting Null Effects to Achieve Variance Reduction in Experiments with One-sided Triggering
Preprint
Abstract
In online experiments where the intervention is only exposed, or "triggered", for a small subset of the population, it is critical to use variance reduction techniques to estimate treatment effects with sufficient precision to inform business decisions. Trigger-dilute analysis is often used in these situations, and reduces the sampling variance of overall intent-to-treat (ITT) effects by an order of magnitude equal to the inverse of the triggering rate; for example, a triggering rate of $5\%$ corresponds to roughly a $20x$ reduction in variance. To apply trigger-dilute analysis, one needs to know experimental subjects' triggering counterfactual statuses, i.e., the counterfactual behavior of subjects under both treatment and control conditions. In this paper, we propose an unbiased ITT estimator with reduced variance applicable for experiments where the triggering counterfactual status is only observed in the treatment group. Our method is based on the efficiency augmentation idea of CUPED and draws upon identification frameworks from the principal stratification and instrumental variables literature. The unbiasedness of our estimation approach relies on a testable assumption that the augmentation term used for covariate adjustment equals zero in expectation. Unlike traditional covariate adjustment or principal score modeling approaches, our estimator can incorporate both pre-experiment and in-experiment observations. We demonstrate through a real-world experiment and simulations that our estimator can remain unbiased and achieve precision improvements as large as if triggering status were fully observed, and in some cases can even outperform trigger-dilute analysis.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- On the implied weights of linear regression for causal inferenceBiometrika, 2022
- Rerandomization and Regression AdjustmentJournal of the Royal Statistical Society Series B: Statistical Methodology, 2019
- Top Challenges from the first Practical Online Controlled Experiments SummitACM SIGKDD Explorations Newsletter, 2019
- XGBoost2016
- Principal Stratification Analysis Using Principal ScoresJournal of the Royal Statistical Society Series B: Statistical Methodology, 2016
- Covariate Balancing Propensity ScoreJournal of the Royal Statistical Society Series B: Statistical Methodology, 2013
- Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational StudiesPolitical Analysis, 2012
- On the use of propensity scores in principal causal effect estimationStatistics in Medicine, 2009
- Controlled experiments on the web: survey and practical guideData Mining and Knowledge Discovery, 2008
- Principal Stratification in Causal InferenceBiometrics, 2002