An empirical study on the usage of mocking frameworks in Apache software foundation
Open Access
- 23 January 2024
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 29 (2) , 1-53
Abstract
Mocking frameworks provide convenient APIs, which create mock objects, manipulate their behavior, and verify their execution, for the purpose of isolating test dependencies in unit testing. This study contributes an in-depth empirical study of whether and how mocking frameworks are used in Apache projects. The key findings and insights of this study include: First, mocking frameworks are widely used in 66% of Apache Java projects, with Mockito, EasyMock, and PowerMock being the top three most popular frameworks. Larger-scale and more recent projects tend to observe a stronger need to use mocking frameworks. This underscores the importance of mocking in practice and related future research. Second, mocking is overall practiced quite selectively in software projects—not all test files use mocking, nor all dependencies of a test target are mocked. It calls for more future research to gain a more systematic understanding of when and what to mock to provide formal guidance to practitioners. On top of this, the intensity of mocking in different projects shows different trends in the projects’ evolution history—implying the compound effects of various factors, such as the pace of a project’s growth, the available resources, time pressure, and priority, etc. This points to an important future research direction in facilitating best mocking practices in software evolution. Furthermore, we revealed the most frequently used APIs in the three most popular frameworks, organized based on the function types. The top five APIs in each functional type of the three mocking frameworks usually take the majority (78% to 100%) of usage in Apache projects. This indicates that developers can focus on these APIs to quickly learn the common usage of these mocking frameworks. We further investigated informal methods of mocking, which do not rely on any mocking framework. These informal mocking methods point to potential sub-optimal mocking practices that could be improved, as well as limitations of existing mocking frameworks. Finally, we conducted a developer survey to collect additional insights regarding the above analysis based on their experience, which complements our analysis based on repository mining. Overall, this study offers practitioners profound empirical knowledge of how mocking frameworks are used in practice and sheds light on future research directions to enhancing mocking in practice.Keywords
This publication has 29 references indexed in Scilit:
- Contract-Based Mocking for Services-Oriented Development2016
- A Survey on Unit Testing Practices and Problems2014
- An Empirical Study on the Usage of Mocking Frameworks in Software Testing2014
- A survey of software testing practices in CanadaJournal of Systems and Software, 2013
- Open source software peer review practices2008
- Software Testing Research: Achievements, Challenges, Dreams2007
- Assessing the Health of Open Source CommunitiesComputer, 2006
- Evolution of Open Source Communities2006
- Mock roles, not objects2004
- Two case studies of open source software developmentACM Transactions on Software Engineering and Methodology, 2002