Comparing the effectiveness of capture and replay against automatic input generation for Android graphical user interface testing
- 16 October 2020
- journal article
- research article
- Published by Wiley in Software Testing, Verification and Reliability
- Vol. 31 (3)
Abstract
Summary: Exploratory testing and fully automated testing tools represent two viable and cheap alternatives to traditional test‐case‐based approaches for graphical user interface (GUI) testing of Android apps. The former can be executed by capture and replay tools that directly translate execution scenarios registered by testers in test cases, without requiring preliminary test‐case design and advanced programming/testing skills. The latter tools are able to test Android GUIs without tester intervention. Even if these two strategies are widely employed, to the best of our knowledge, no empirical investigation has been performed to compare their performance and obtain useful insights for a project manager to establish an effective testing strategy. In this paper, we present two experiments we carried out to compare the effectiveness of exploratory testing approaches using a capture and replay tool (Robotium Recorder) against three freely available automatic testing tools (AndroidRipper, Sapienz, and Google Robo). The first experiment involved 20 computer engineering students who were asked to record testing executions, under strict temporal limits and no access to the source code. Results were slightly better than those of fully automated tools, but not in a conclusive way. In the second experiment, the same students were asked to improve the achieved testing coverage by exploiting the source code and the coverage obtained in the previous tests, without strict temporal constraints. The results of this second experiment showed that students outperformed the automated tools especially for long/complex execution scenarios. The obtained findings provide useful indications for deciding testing strategies that combine manual exploratory testing and automated testing.Keywords
This publication has 45 references indexed in Scilit:
- Mobile Application Verification: A Systematic Mapping Study2015
- Are Students Representatives of Professionals in Software Engineering Experiments?2015
- Understanding the Test Automation Culture of App Developers2015
- Conceptualization and Evaluation of Component-Based Testing Unified with Visual GUI Testing: An Empirical Study2015
- Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosystem2015
- Automatic testing of GUI‐based applicationsSoftware Testing, Verification and Reliability, 2014
- Visual vs. DOM-Based Web Locators: An Empirical Study2014
- Guided GUI testing of android apps with minimal restart and approximate learning2013
- Targeted and depth-first exploration for systematic testing of android appsACM SIGPLAN Notices, 2013
- Issues in using students in empirical studies in software engineering education2004