Automata-Based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android
Open Access
- 30 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 592-604
Abstract
Benchmarking software testing tools against known bugs is a classic approach to evaluating the tools’ bug finding abilities. However, this approach is difficult to give some clues on the tool-missed bugs to aid diagnosing the testing tools. As a result, heavy and ad hoc manual analysis is needed. In this work, in the setting of GUI testing for Android apps, we introduce an automata-based trace analysis approach to tackling the key challenge of manual analysis, i.e., how to analyze the lengthy event traces generated by a testing tool against a missed bug to find the clues. Our key idea is that, we model a bug in the form of a finite automaton which captures its bug-triggering traces; and match the event traces generated by the testing tool (which misses this bug) against this automaton to obtain the clues. Specifically, the clues are presented in the form of three designated automata-based coverage values. We apply our approach to enhance Themis, a representative benchmark suite for Android, to aid diagnosing GUI testing tools. Our extensive evaluation on nine state-of-the-art GUI testing tools and the involvement with several tool developers shows that our approach is feasible and useful. Our approach enables Themis+ (the enhanced benchmark suite) to provide the clues on the tool-missed bugs, and all the Themis+’s clues are identical or useful, compared to the manual analysis results of tool developers. Moreover, the clues have helped find several tool weaknesses, which were unknown or unclear before. Based on the clues, two actively-developing industrial testing tools in our study have quickly made several optimizations and demonstrated their improved bug finding abilities. All the tool developers give positive feedback on the usefulness and usability of Themis+’s clues. Themis+ is available at https://github.com/DDroid-Android/home.Keywords
This publication has 48 references indexed in Scilit:
- Sapienz: multi-objective automated testing for Android applications2016
- RV-Android: Efficient Parametric Android Runtime Verification, a Brief Tutorial2015
- Automated Test Input Generation for Android: Are We There Yet? (E)2015
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs2014
- Guided GUI testing of android apps with minimal restart and approximate learning2013
- Runtime Verification and Enforcement for Android Applications with RV-DroidLecture Notes in Computer Science, 2013
- An overview of the MOP runtime verification frameworkInternational Journal on Software Tools for Technology Transfer, 2011
- Experiences from Conducting Semi-structured Interviews in Empirical Software Engineering Research2005
- Automata-based verification of temporal properties on running programs2005
- Software unit test coverage and adequacyACM Computing Surveys, 1997