The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
Preprint
Abstract
Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A3Test: Assertion-Augmented Automated Test case generationInformation and Software Technology, 2024
- Evaluating and Improving ChatGPT for Unit Test GenerationProceedings of the ACM on Software Engineering, 2024
- Exploring and Unleashing the Power of Large Language Models in Automated Code TranslationProceedings of the ACM on Software Engineering, 2024
- Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLMProceedings of the ACM on Software Engineering, 2024
- Lost in the Middle: How Language Models Use Long ContextsTransactions of the Association for Computational Linguistics, 2024
- Software Testing With Large Language Models: Survey, Landscape, and VisionIEEE Transactions on Software Engineering, 2024
- Code Structure–Guided Transformer for Source Code SummarizationACM Transactions on Software Engineering and Methodology, 2023
- Benefits and limitations of automated software testing: Systematic literature review and practitioner survey2012
- A survey of unit testing practicesIEEE Software, 2006
- Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous SpacesJournal of Global Optimization, 1997