Towards Autonomous Testing Agents via Conversational Large Language Models
Authors:
- 11 September 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1688-1693
Abstract
Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized “hallucination” of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Towards developer-centered automatic program repair: findings from Bloomberg2022
- How can manual testing processes be optimized? developer survey, optimization guidelines, and case studies2021
- Boundary Value Exploration for Software Analysis2020
- An experience report on applying software testing academic results in industry: we need usable automated test generationEmpirical Software Engineering, 2017
- Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and PracticesIEEE Access, 2017
- Test Set Diameter: Quantifying the Diversity of Sets of Test Cases2016
- Mutation-driven generation of unit tests and oracles2010
- Adaptive Random Testing: The ART of test case diversityJournal of Systems and Software, 2010
- A Case for Test-Code Generation in Model-Driven SystemsLecture Notes in Computer Science, 2003