Studying Test Annotation Maintenance in the Wild
Authors:
- 1 May 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 62-73
Abstract
Since the introduction of annotations in Java 5, the majority of testing frameworks, such as JUnit, TestNG, and Mockito, have adopted annotations in their core design. This adoption affected the testing practices in every step of the test lifecycle, from fixture setup and test execution to fixture teardown. Despite the importance of test annotations, most research on test maintenance has mainly focused on test code quality and test assertions. As a result, there is little empirical evidence on the evolution and maintenance of test annotations. To fill this gap, we perform the first fine-grained empirical study on annotation changes. We developed a tool to mine 82,810 commits and detect 23,936 instances of test annotation changes from 12 open-source Java projects. Our main findings are: (1) Test annotation changes are more frequent than rename and type change refactorings. (2) We recover various migration efforts within the same testing framework or between different frameworks by analyzing common annotation replacement patterns. (3) We create a taxonomy by manually inspecting and classifying a sample of 368 test annotation changes and documenting the motivations driving these changes. Finally, we present a list of actionable implications for developers, researchers, and framework designers.Keywords
This publication has 38 references indexed in Scilit:
- Practical regression test selection with dynamic file dependencies2015
- An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes2015
- Test Code Quality and Its Relation to Issue Handling PerformanceIEEE Transactions on Software Engineering, 2014
- Are test smells really harmful? An empirical studyEmpirical Software Engineering, 2014
- Fault localization prioritizationACM Transactions on Software Engineering and Methodology, 2013
- Adoption and use of Java genericsEmpirical Software Engineering, 2012
- Static test case prioritization using topic modelsEmpirical Software Engineering, 2012
- Studying the co-evolution of production and test code in open source and industrial developer test processes through repository miningEmpirical Software Engineering, 2010
- Prioritizing test cases for regression testingIEEE Transactions on Software Engineering, 2001
- A safe, efficient regression test selection techniqueACM Transactions on Software Engineering and Methodology, 1997