Test smells 20 years later: detectability, validity, and reliability
Open Access
- 20 September 2022
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 27 (7) , 1-40
Abstract
Test smells aim to capture design issues in test code that reduces its maintainability. These have been extensively studied and generally found quite prevalent in both human-written and automatically generated test-cases. However, most evidence of prevalence is based on specific static detection rules. Although those are based on the original, conceptual definitions of the various test smells, recent empirical studies indicate that developers perceive warnings raised by detection tools as overly strict and non-representative of the maintainability and quality of test suites. This leads us to re-assess test smell detection tools’ detection accuracy and investigate the prevalence and detectability of test smells more broadly. Specifically, we construct a hand-annotated dataset spanning hundreds of test suites both written by developers and generated by two test generation tools (EvoSuiteandJTExpert) and performed a multi-stage, cross-validated manual analysis to identify the presence of six types of test smells in these. We then use this manual labeling to benchmark the performance and external validity of two test smell detection tools—one widely used in prior work and one recently introduced with the express goal to match developer perceptions of test smells. Our results primarily show that the current vocabulary of test smells is highly mismatched to real concerns: multiple smells were ubiquitous on developer-written tests but virtually never correlated with semantic or maintainability flaws; machine-generated tests actually often scored better, but in reality, suffered from a host of problems not well-captured by current test smells. Current test smell detection strategies poorly characterized the issues in these automatically generated test suites; in particular, the older tool’s detection strategies misclassified over 70% of test smells, both missing real instances (false negatives) and marking many smell-free tests as smelly (false positives). We identify common patterns in these tests that can be used to improve the tools, refine and update the definition of certain test smells, and highlight as of yet uncharacterized issues. Our findings suggest the need for (i) more appropriate metrics to match development practice, (ii) more accurate detection strategies to be evaluated primarily in industrial contexts.Keywords
Funding Information
- H2020 European Research Council (957254-COSMOS)
- Engineering and Physical Sciences Research Council (EP/N023978/2)
This publication has 59 references indexed in Scilit:
- Combining Multiple Coverage Criteria in Search-Based Unit Test Generation2015
- Defect prediction as a multiobjective optimization problemSoftware Testing, Verification and Reliability, 2015
- Instance Generator and Problem Representation to Improve Object Oriented Code CoverageIEEE Transactions on Software Engineering, 2014
- Are test smells really harmful? An empirical studyEmpirical Software Engineering, 2014
- Achieving scalable mutation-based generation of whole test suitesEmpirical Software Engineering, 2014
- 1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuiteEmpirical Software Engineering, 2013
- Parameter tuning or default values? An empirical investigation in search-based software engineeringEmpirical Software Engineering, 2013
- Regression testing minimization, selection and prioritization: a surveySoftware Testing, Verification and Reliability, 2012
- Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking2006
- Evolutionary testing of classesACM SIGSOFT Software Engineering Notes, 2004