Rotten Green Tests
Authors:
- 1 May 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 500-511
Abstract
Unit tests are a tenant of agile programming methodologies, and are widely used to improve code quality and prevent code regression. A green (passing) test is usually taken as a robust sign that the code under test is valid. However, some green tests contain assertions that are never executed. We call such tests Rotten Green Tests. Rotten Green Tests represent a case worse than a broken test: they report that the code under test is valid, but in fact do not test that validity. We describe an approach to identify rotten green tests by combining simple static and dynamic call-site analyses. Our approach takes into account test helper methods, inherited helpers, and trait compositions, and has been implemented in a tool called DrTest. DrTest reports no false negatives, yet it still reports some false positives due to conditional use or multiple test contexts. Using DrTest we conducted an empirical evaluation of 19,905 real test cases in mature projects of the Pharo ecosystem. The results of the evaluation show that the tool is effective; it detected 294 tests as rotten—green tests that contain assertions that are not executed. Some rotten tests have been “sleeping” in Pharo for at least 5 years.Keywords
This publication has 30 references indexed in Scilit:
- Empirically Detecting False Test Alarms Using Association Rules2015
- Checked coverage: an indicator for oracle qualitySoftware Testing, Verification and Reliability, 2013
- Reusing and Composing Tests with Traits2009
- Rule-based Assessment of Test Quality.The Journal of Object Technology, 2007
- TraitsACM Transactions on Programming Languages and Systems, 2006
- Automatic Test Case Optimization: A Bacteriologic AlgorithmIEEE Software, 2005
- JCrasher: an automatic robustness tester for JavaSoftware: Practice and Experience, 2004
- Demand-Driven Type Inference with Subgoal Pruning: Trading Precision for Scalability2004
- Partial behavioral reflection2003
- The Test Automation Manifesto2003