Adaptive random testing
Authors:
Open Access
- 17 July 2011
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 265-275
Abstract
Adaptive Random Testing (ART) has been proposed as an enhancement to random testing, based on assumptions on how failing test cases are distributed in the input domain. The main assumption is that failing test cases are usually grouped into contiguous regions. Many papers have been published in which ART has been described as an effective alternative to random testing when using the average number of test case executions needed to find a failure (F-measure). But all the work in the literature is based either on simulations or case studies with unreasonably high failure rates. In this paper, we report on the largest empirical analysis of ART in the literature, in which 3727 mutated programs and nearly ten trillion test cases were used. Results show that ART is highly inefficient even on trivial problems when accounting for distance calculations among test cases, to an extent that probably prevents its practical use in most situations. For example, on the infamous Triangle Classification program, random testing finds failures in few milliseconds whereas ART execution time is prohibitive. Even when assuming a small, fixed size test set and looking at the probability of failure (P-measure), ART only fares slightly better than random testing, which is not sufficient to make it applicable in realistic conditions. We provide precise explanations of this phenomenon based on rigorous empirical analyses. For the simpler case of single-dimension input domains, we also perform formal analyses to support our claim that ART is of little use in most situations, unless drastic enhancements are developed. Such analyses help us explain some of the empirical results and identify the components of ART that need to be improved to make it a viable option in practice.Keywords
This publication has 31 references indexed in Scilit:
- Adaptive Random Testing: The ART of test case diversityJournal of Systems and Software, 2010
- A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs2009
- Adaptive Random Test Case Prioritization2009
- Adaptive random testing based on distribution metricsJournal of Systems and Software, 2009
- Distributing test cases more evenly in adaptive random testingJournal of Systems and Software, 2008
- Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parametersSoftware Quality Journal, 2008
- Adaptive Random Testing2004
- Empirical comparison of regression test selection algorithmsJournal of Systems and Software, 2001
- NASA Software failure characterization experimentsReliability Engineering & System Safety, 1991
- Data diversity: an approach to software fault toleranceIEEE Transactions on Computers, 1988