The integration of machine learning into automated test generation: A systematic mapping study
Authors:
Open Access
- 2 May 2023
- journal article
- research article
- Published by Wiley in Software Testing, Verification and Reliability
- Vol. 33 (4)
Abstract
Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.Keywords
All Related Versions
Funding Information
- Vetenskapsrådet (2019‐05275)
This publication has 129 references indexed in Scilit:
- Guided GUI testing of android apps with minimal restart and approximate learningACM SIGPLAN Notices, 2013
- An orchestrated survey of methodologies for automated software test case generationJournal of Systems and Software, 2013
- Coverage-Directed Test Generation Automated by Machine Learning -- A ReviewACM Transactions on Design Automation of Electronic Systems, 2012
- Radial basis function neural network based approach to test oracleACM SIGSOFT Software Engineering Notes, 2011
- Artificial neural networks as multi-networks automated test oracleAutomated Software Engineering, 2011
- An automated framework for software test oracleInformation and Software Technology, 2011
- Neural Networks Based Automated Test Oracle for Software Testing2006
- Online Testing with Reinforcement LearningLecture Notes in Computer Science, 2006
- Using a neural network in the software testing processInternational Journal of Intelligent Systems, 2002
- Test case generation by means of learning techniquesACM SIGSOFT Software Engineering Notes, 1993