Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster
Open Access
- 6 March 2023
- journal article
- research article
- Published by Springer Nature in Software Quality Journal
- Vol. 31 (3) , 947-990
Abstract
Research in software testing often involves the development of software prototypes. Like any piece of software, there are challenges in the development, use and verification of such tools. However, some challenges are rather specific to this problem domain. For example, often these tools are developed by PhD students straight out of bachelor/master degrees, possibly lacking any industrial experience in software development. Prototype tools are used to carry out empirical studies, possibly studying different parameters of novel designed algorithms. Software scaffolding is needed to run large sets of experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs to be taken to deal with their randomness, which further complicates their verification. The aforementioned represent some of the challenges we have identified for this domain. In this paper, we report on our experience in building the open-source EvoMaster tool, which aims at system-level test case generation for enterprise applications. Many of the challenges we faced would be common to any researcher needing to build software testing tool prototypes. Therefore, one goal is that our shared experience here will boost the research community, by providing concrete solutions to many development challenges in the building of such kind of research prototypes. Ultimately, this will lead to increase the impact of scientific research on industrial practice.Keywords
Funding Information
- European Research Council (864972, 864972, 864972, 864972, 864972)
- Secretaría de Ciencia y Técnica, Universidad de Buenos Aires (UBACyT 2020 20020190100233BA)
- Agencia Nacional de Promoción de la Investigación, el Desarrollo Tecnológico y la Innovación (PICT-2019-01793)
- Kristiania University College
This publication has 79 references indexed in Scilit:
- On the techniques we create, the tools we build, and their misalignments2016
- Parameter tuning or default values? An empirical investigation in search-based software engineeringEmpirical Software Engineering, 2013
- A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineeringSoftware Testing, Verification and Reliability, 2012
- Search-based software engineeringACM Computing Surveys, 2012
- EvoSuite2011
- A survey of combinatorial testingACM Computing Surveys, 2011
- A few billion lines of code laterCommunications of the ACM, 2010
- Randoop2007
- Software Testing Research: Achievements, Challenges, Dreams2007
- Automated software test data generationIEEE Transactions on Software Engineering, 1990