On Post-Selection Inference in A/B Tests
Authors:
- Alex DengADAlex Deng
- Yicheng LiYLYicheng Li
- Jiannan LuJLJiannan Lu
- Vivek RamamurthyVRVivek Ramamurthy
Preprint
Abstract
When interpreting A/B tests, we typically focus only on the statistically significant results and take them by face value. This practice, termed post-selection inference in the statistical literature, may negatively affect both point estimation and uncertainty quantification, and therefore hinder trustworthy decision making in A/B testing. To address this issue, in this paper we explore two seemingly unrelated paths, one based on supervised machine learning and the other on empirical Bayes, and propose post-selection inferential approaches that combine the strengths of both. Through large-scale simulated and empirical examples, we demonstrate that our proposed methodologies stand out among other existing ones in both reducing post-selection biases and improving confidence interval coverage rates, and discuss how they can be conveniently adjusted to real-life scenarios.Keywords
All Related Versions
- Version 1, 2019-10-09, ArXiv
- Version 2, 2021-05-30, ArXiv
- Published version: , 2743.
This publication has 15 references indexed in Scilit:
- Top Challenges from the first Practical Online Controlled Experiments SummitACM SIGKDD Explorations Newsletter, 2019
- The A/B Testing Problem2018
- Post‐selection point and interval estimation of signal sizes in Gaussian samplesCanadian Journal of Statistics, 2017
- Demystifying the bias from selective inference: A revisit to Dawid’s treatment selection problemStatistics & Probability Letters, 2016
- Bayesian Reasoning and Machine Learning2012
- Simultaneous and selective inference: Current successes and future challengesBiometrical Journal, 2010
- Controlled experiments on the web: survey and practical guideData Mining and Knowledge Discovery, 2008
- Asymptotic Theory of Statistics and Probability2008
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B: Statistical Methodology, 1995
- Exact and Approximate Posterior Moments for a Normal Location ParameterJournal of the Royal Statistical Society Series B: Statistical Methodology, 1992