On Post-selection Inference in A/B Testing
Open Access
- 14 August 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2743-2752
Abstract
When interpreting A/B tests, we typically focus only on the statistically significant results and take them by face value. This practice, termed post-selection inference in the statistical literature, may negatively affect both point estimation and uncertainty quantification, and therefore hinder trustworthy decision making in A/B testing. To address this issue, in this paper we explore two seemingly unrelated paths, one based on supervised machine learning and the other on empirical Bayes, and propose post-selection inferential approaches that combine the strengths of both. Through large-scale simulated and empirical examples, we demonstrate that our proposed methodologies stand out among other existing ones in both reducing post-selection biases and improving confidence interval coverage rates, and discuss how they can be conveniently adjusted to real-life scenarios.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- From Infrastructure to Culture2015
- Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments2015
- Online controlled experiments at large scale2013
- Non-negative least squares for high-dimensional linear models: Consistency and sparse recovery without regularizationElectronic Journal of Statistics, 2013
- SURE Estimates for a Heteroscedastic Hierarchical ModelJournal of the American Statistical Association, 2012
- Simultaneous and selective inference: Current successes and future challengesBiometrical Journal, 2010
- Overlapping experiment infrastructure2010
- Controlled experiments on the web: survey and practical guideData Mining and Knowledge Discovery, 2008
- Adapting to Unknown Smoothness via Wavelet ShrinkageJournal of the American Statistical Association, 1995
- THE PROBABLE ERROR OF A MEANBiometrika, 1908