Mobile GUI Testing Fragility: A Study on Open-Source Android Applications
- 30 September 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Reliability
- Vol. 68 (1) , 67-90
Abstract
Android applications do not seem to be tested as thoroughly as desktop ones. In particular, graphical user interface (GUI) testing appears generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e., GUI test classes failing or needing updates due to even minor modifications in the GUI or in the application under test. The objective of our study is to estimate the adoption of GUI testing frameworks among Android open-source applications, the quantity of modifications needed to keep test classes up to date, and their amount due to GUI test fragility. We introduce a set of 21 metrics to measure the adoption of testing tools and the evolution of test classes and test methods, and to estimate the fragility of test suites. We computed our metrics for six GUI testing frameworks, none of which achieved a significant adoption among Android projects hosted on GitHub. When present, GUI test methods associated with the considered tools are modified often, and a relevant portion (70% on average) of those modifications is induced by GUI-related fragilities. On average, for the projects considered, more than 7% of the total modified lines of code between consecutive releases belong to test classes developed with the analyzed testing frameworks. The measured percentage was higher on average than the one required by other generic test code, based on the JUnit testing framework. Fragility of GUI tests constitutes a relevant concern, probably an obstacle for developers to adopt test automation. This first evaluation of the fragility of Android scripted GUI testing can constitute a benchmark for developers and testers leveraging the analyzed test tools and the basis for the definition of a taxonomy of fragility causes and guidelines to mitigate the issue.Keywords
Funding Information
- TIM
This publication has 41 references indexed in Scilit:
- Enabling Testing of Android Apps2015
- Understanding the Test Automation Culture of App Developers2015
- Software test-code engineering: A systematic mappingInformation and Software Technology, 2015
- Visual GUI testing in practice: challenges, problemsand limitationsEmpirical Software Engineering, 2014
- Visual vs. DOM-Based Web Locators: An Empirical Study2014
- Is lines of code a good measure of effort in effort-aware models?Information and Software Technology, 2013
- Guided GUI testing of android apps with minimal restart and approximate learningACM SIGPLAN Notices, 2013
- Testing Android Mobile Applications: Challenges, Strategies, and Approaches2013
- Testing android apps through symbolic executionACM SIGSOFT Software Engineering Notes, 2012
- Towards Unit Testing of User Interface Code for Android Mobile ApplicationsCommunications in Computer and Information Science, 2011