GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
Open Access
- 3 November 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (11) , 1-32
Abstract
With data-driven development now widely adopted in the software industry, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands considerable resources for experimental design, implementation, and production deployment and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring multiple weeks of data collection. To tackle these issues, a growing research area known as off-policy evaluation (OPE) (offline A/B testing) has gained significant attention. Its goal is to assess new technologies offline by leveraging previously collected logged data. OPE is also a fundamental problem in reinforcement learning (RL) and is important in areas where online A/B testing is expensive or risky, such as healthcare, recommender systems, education, dialog systems, and robotics. Despite recent advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can be leveraged to automatically optimize OPE implementations. In this work, we propose GrowthHacker , a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the resulting metrics to guide subsequent rounds of optimization. We evaluate methods on Open Bandit Pipeline and Scope-RL, and we develop a two_agent framework that addresses limitations of existing agent frameworks while reducing system complexity. Across both libraries, two_agent demonstrates the highest reliability (98.1–100% success rate) and the highest positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated “growth hackers” to continuously improve OPE systems, with implications for scaling data-driven decision-making in production environments where manual optimization is expensive. Our replication package is available at https://doi.org/10.5281/zenodo.17496869 and https://github.com/jie-jw-wu/ope-agent .Keywords
All Related Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-05175)
This publication has 29 references indexed in Scilit:
- Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models2025
- A multi-objective evolutionary approach towards automated online controlled experimentsJournal of Systems and Software, 2023
- Comparison of multi-criteria decision-making methods for online controlled experiments in a launch decision-making frameworkInformation and Software Technology, 2022
- Controlled experimentation in continuous experimentation: Knowledge and challengesInformation and Software Technology, 2021
- Understanding and improving the quality and reproducibility of Jupyter notebooksEmpirical Software Engineering, 2021
- The Online Controlled Experiment LifecycleIEEE Software, 2018
- The RIGHT model for Continuous ExperimentationJournal of Systems and Software, 2017
- Continuous software engineering: A roadmap and agendaJournal of Systems and Software, 2017
- Development and Deployment at FacebookIEEE Internet Computing, 2013
- A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each2012