AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement Engineering
Author:
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 472-476
Abstract
Software companies have widely used online A/B testing to evaluate the impact of a new technology by offering it to groups of users and comparing it against the unmodified product. However, running online A/B testing needs not only efforts in design, implementation, and stakeholders’ approval to be served in production but also several weeks to collect the data in iterations. To address these issues, a recently emerging topic, called offline A/B testing, is getting increasing attention, intending to conduct the offline evaluation of new technologies by estimating historical logged data. Although this approach is promising due to lower implementation effort, faster turnaround time, and no potential user harm, for it to be effectively prioritized as requirements in practice, several limitations need to be addressed, including its discrepancy with online A/B test results, and lack of systematic updates on varying data and parameters. In response, in this vision paper, I introduce AutoOffAB, an idea to automatically run variants of offline A/B testing against recent logging and update the offline evaluation results, which are used to make decisions on requirements more reliably and systematically.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Controlled experimentation in continuous experimentation: Knowledge and challengesInformation and Software Technology, 2021
- The Online Controlled Experiment LifecycleIEEE Software, 2018
- Offline A/B Testing for Recommender Systems2018
- The RIGHT model for Continuous ExperimentationJournal of Systems and Software, 2017
- Continuous software engineering: A roadmap and agendaJournal of Systems and Software, 2017
- Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement2016
- From Infrastructure to Culture2015
- Development and Deployment at FacebookIEEE Internet Computing, 2013
- Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms2011
- Overlapping experiment infrastructure2010