Data-Driven Metric Development for Online Controlled Experiments
Authors:
Open Access
- 13 August 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 77-86
Abstract
Online controlled experiments, also called A/B testing, have been established as the mantra for data-driven decision making in many web-facing companies. In recent years, there are emerging research works focusing on building the platform and scaling it up, best practices and lessons learned to obtain trustworthy results, and experiment design techniques and various issues related to statistical inference and testing. However, despite playing a central role in online controlled experiments, there is little published work on treating metric development itself as a data-driven process. In this paper, we focus on the topic of how to develop meaningful and useful metrics for online services in their online experiments, and show how data-driven techniques and criteria can be applied in metric development process. In particular, we emphasize two fundamental qualities for the goal metrics (or Overall Evaluation Criteria) of any online service: directionality and sensitivity. We share lessons on why these two qualities are critical, how to measure these two qualities of metrics of interest, how to develop metrics with clear directionality and high sensitivity by using approaches based on user behavior models and data-driven calibration, and how to choose the right goal metrics for the entire online services.Keywords
This publication has 28 references indexed in Scilit:
- Focusing on the Long-term2015
- Context-aware web search abandonment prediction2014
- A Scalable Bootstrap for Massive DataJournal of the Royal Statistical Society Series B: Statistical Methodology, 2014
- Trustworthy online controlled experiments2012
- Key lessons learned building recommender systems for large-scale social networks2012
- Find it if you can2011
- Unexpected results in online controlled experimentsACM SIGKDD Explorations Newsletter, 2011
- Overlapping experiment infrastructure2010
- Controlled experiments on the web: survey and practical guideData Mining and Knowledge Discovery, 2008
- Evaluating implicit measures to improve web searchACM Transactions on Information Systems, 2005