Trustworthy Online Controlled Experiments
Authors:
Top Cited Papers
- 13 March 2020
- book
- Published by Cambridge University Press (CUP)
Abstract
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments • Define key metrics and ideally an Overall Evaluation Criterion • Test for trustworthiness of the results and alert experimenters to violated assumptions • Build a scalable platform that lowers the marginal cost of experiments close to zero • Avoid pitfalls like carryover effects and Twyman's law • Understand how statistical issues play out in practice.Keywords
This publication has 115 references indexed in Scilit:
- Large-scale validation and analysis of interleaved search evaluationACM Transactions on Information Systems, 2012
- Regulating the Production of Knowledge: Research Risk-Benefit Analysis and the Heterogeneity ProblemSSRN Electronic Journal, 2012
- On the accuracy of the Gaussian approximationDoklady Mathematics, 2009
- Controlled experiments on the web: survey and practical guideData Mining and Knowledge Discovery, 2008
- Rethinking Timing of First Sex and DelinquencyJournal of Youth and Adolescence, 2007
- Contradicted and Initially Stronger Effects in Highly Cited Clinical ResearchJAMA, 2005
- The Progress of Experiment: Science and Therapeutic Reform in the United States, 1900-1990The American Historical Review, 1999
- Formal mode of statistical inference for causal effectsJournal of Statistical Planning and Inference, 1990
- Equipoise and the Ethics of Clinical ResearchNew England Journal of Medicine, 1987
- The central role of the propensity score in observational studies for causal effectsBiometrika, 1983