Ayanda Patrick Herlihy
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Ayanda Patrick Herlihy has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Ahmad Saleem Mirza
The University of British Columbia Okanagan, Kelowna, British Columbia, Canada
1 shared publication - Ali Afoud
The University of British Columbia Okanagan, Kelowna, British Columbia, Canada
1 shared publication - Fatemeh Hendijani Fard
Department of Computer Science, University of British Columbia, Kelowna, Canada
1 shared publication - Jie Jw Wu
University of British Columbia, Kelowna, Canada
1 shared publication
Latest Publications
- Open Access
- 3 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (11) , 1-32
GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
With data-driven development now widely adopted in the software industry, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands considerable resources for experimental design, implementation, and production deployment and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring multiple weeks of data collection. To tackle these issues, a growing research area known as off-policy evaluation (OPE) (offline A/B testing) has gained significant attention. Its goal is to assess new technologies offline by leveraging previously collected logged data. OPE is also a fundamental problem in reinforcement learning (RL) and is important in areas where online A/B testing is expensive or risky, such as healthcare, recommender systems, education, dialog systems, and robotics. Despite recent advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can be leveraged to automatically optimize OPE implementations. In this work, we propose GrowthHacker , a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the resulting metrics to guide subsequent rounds of optimization. We evaluate methods on Open Bandit Pipeline and Scope-RL, and we develop a two_agent framework that addresses limitations of existing agent frameworks while reducing system complexity. Across both libraries, two_agent demonstrates the highest reliability (98.1–100% success rate) and the highest positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated “growth hackers” to continuously improve OPE systems, with implications for scaling data-driven decision-making in production environments where manual optimization is expensive. Our replication package is available at https://doi.org/10.5281/zenodo.17496869 and https://github.com/jie-jw-wu/ope-agent .
- 3 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (11) , 1-32
Open AccessGrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
With data-driven development now widely adopted in the software industry, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands considerable resources for experimental design, implementation, and production deployment and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring multiple weeks of data collection. To tackle these issues, a growing research area known as off-policy evaluation (OPE) (offline A/B testing) has gained significant attention. Its goal is to assess new technologies offline by leveraging previously collected logged data. OPE is also a fundamental problem in reinforcement learning (RL) and is important in areas where online A/B testing is expensive or risky, such as healthcare, recommender systems, education, dialog systems, and robotics. Despite recent advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can be leveraged to automatically optimize OPE implementations. In this work, we propose GrowthHacker , a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the resulting metrics to guide subsequent rounds of optimization. We evaluate methods on Open Bandit Pipeline and Scope-RL, and we develop a two_agent framework that addresses limitations of existing agent frameworks while reducing system complexity. Across both libraries, two_agent demonstrates the highest reliability (98.1–100% success rate) and the highest positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated “growth hackers” to continuously improve OPE systems, with implications for scaling data-driven decision-making in production environments where manual optimization is expensive. Our replication package is available at https://doi.org/10.5281/zenodo.17496869 and https://github.com/jie-jw-wu/ope-agent .