Test Set Diameter: Quantifying the Diversity of Sets of Test Cases
Authors:
- Robert FeldtRFRobert Feldt
- Simon PouldingSPSimon Poulding
- David ClarkDCDavid Clark
- Shin YooSYShin Yoo
- 1 April 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 223-233
Abstract
A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. A key advantage is that TSDm is a universal measure of diversity and so can be applied to any test set regardless of data type of the test inputs (and, moreover, to other test-related data such as execution traces). But this universality comes at the cost of greater computational effort compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- An initial industrial evaluation of interactive search-based testing for embedded softwareApplied Soft Computing, 2015
- Diversity oriented test data generation using metaheuristic search techniquesInformation Sciences, 2014
- Supporting Software Decision Meetings: Heatmaps for Visualising Test and Code Measurements2013
- Measuring and Improving Latency to Avoid Test Suite Wear Out2009
- Test diversityInformation and Software Technology, 2006
- Object distance and its application to adaptive random testing of object-oriented programs2006
- A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases2005
- Clustering by CompressionIEEE Transactions on Information Theory, 2005
- Algorithmic Clustering of Music Based on String CompressionComputer Music Journal, 2004
- A history-based test prioritization technique for regression testing in resource constrained environments2002