(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs
Open Access
- 14 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 166-171
Abstract
Large Language Models (LLMs) are increasingly integrated into software applications. Downstream application developers often access LLMs through APIs provided as a service. However, LLM APIs are often updated silently and scheduled to be deprecated, forcing users to continuously adapt to evolving models. This can cause performance regression and affect prompt design choices, as evidenced by our case study on toxicity detection. Based on our case study, we emphasize the need for and re-examine the concept of regression testing for evolving LLM APIs. We argue that regression testing LLMs requires fundamental changes to traditional testing approaches, due to different correctness notions, prompting brittleness, and non-determinism in LLM APIs.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning2023
- "Did you miss my comment or what?"2022
- Improving machine translation systems via isotopic replacement2022
- Beware the evolving ‘intelligent’ web service! an integration architecture tactic to guard AI-first components2020
- Automatic testing and improvement of machine translation2020
- Machine Learning in the AWS Cloud2019
- Losing Confidence in Quality: Unspoken Evolution of Computer Vision Services2019
- Bridging the gap between ML solutions and their business requirements using feature interactions2019
- ModelTracker2015
- How Does Web Service API Evolution Affect Clients?2013