Assessing and generating test sets in terms of behavioural adequacy
Open Access
- 20 March 2015
- journal article
- research article
- Published by Wiley in Software Testing, Verification and Reliability
- Vol. 25 (8) , 749-780
Abstract
Summary: Identifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well‐established problem. This is traditionally addressed with syntactic adequacy metrics (e.g. branch coverage), but these can be impractical and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea ofbehavioural coverage: If it is possible to infer an accurate model of a system from its test executions, then the test set can be deemed to be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task) and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. This paper presents a practical approach to incorporate behavioural coverage. OurBESTESTapproach (1) enables the use of machine learning algorithms to augment standard syntactic testing approaches and (2) shows how search‐based testing techniques can be applied to generate test sets with respect to this criterion. An empirical study on a selection of Java units demonstrates that test sets with higher behavioural coverage significantly outperform current baseline test criteria in terms of detected faults. © 2015 The Authors.Software Testing, Verification and Reliabilitypublished by John Wiley & Sons, Ltd.Keywords
Funding Information
- Engineering and Physical Sciences Research Council (EP/K030353/1)
This publication has 47 references indexed in Scilit:
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential ImpactEmpirical Software Engineering, 2005
- Search‐based software test data generation: a surveySoftware Testing, Verification and Reliability, 2004
- Dynamically discovering likely program invariants to support program evolutionIEEE Transactions on Software Engineering, 2001
- Approximate testing and its relationship to learningTheoretical Computer Science, 1997
- The Lack of A Priori Distinctions Between Learning AlgorithmsNeural Computation, 1996
- Testing by means of inductive program learningACM Transactions on Software Engineering and Methodology, 1996
- Inductive inference and software testingSoftware Testing, Verification and Reliability, 1992
- Assessing Test Data Adequacy through Program InferenceACM Transactions on Programming Languages and Systems, 1983
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968