Seeing is Believing: Vision-Driven Non-Crash Functional Bug Detection for Mobile Apps
- 26 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51 (12) , 3452-3466
Abstract
Mobile app GUI (Graphical User Interface) pages now contain rich visual information, with the visual semantics of each page helping users understand the application logic. However, these complex visual and functional logics present new challenges to software testing. Existing automated GUI testing methods, constrained by the lack of reliable testing oracles, are limited to detecting crash bugs with obvious abnormal signals. Consequently, many non-crash functional bugs, ranging from unexpected behaviors to logical errors, often evade detection by current techniques. While these non-crash functional bugs can exhibit visual cues that serve as potential testing oracles, they often entail a sequence of screenshots, and detecting them necessitates an understanding of the operational logic among GUI page transitions, which is challenging traditional techniques. Considering the remarkable performance of Multimodal Large Language Models (MLLM) in visual and language understanding, this paper proposes VisionDroid, a novel vision-driven, multi-agent collaborative automated GUI testing approach for detecting non-crash functional bugs. It comprises three agents: Explorer, Monitor, and Detector, to guide the exploration, oversee the testing progress, and spot issues.We also address several challenges,i.e., aligning visual and textual information for MLLM input, achieving functionality-oriented exploration, and inferring test oracles for non-crash bugs, to enhance the performance of functionality bug detection. We evaluate VisionDroid on 590 non-crash bugs and compare it with 12 baselines, it can achieve more than 14%-112% and 108%-147% boost in average recall and precision compared with the best baseline. The ablation study further proves the contribution of each module. Moreover, VisionDroid identifies 43 unknown bugs on Google Play, of which 31 have been fixed.Keywords
Funding Information
- National Natural Science Foundation of China (62402483)
- National Natural Science Foundation of China (62232016)
- National Natural Science Foundation of China (62072442)
- National Natural Science Foundation of China (62272445)
- National Natural Science Foundation of China (62402484)
- Basic Research Program of ISCAS (ISCAS-JCZD-202304)
- Major Program of ISCAS (ISCAS-ZD-202302)
- Major Program of ISCAS (ISCAS-ZD-202401)
- Innovation Team 2024 ISCAS (2024-66)
This publication has 48 references indexed in Scilit:
- A Study of Using Multimodal LLMs for Non-Crash Functional Bug Detection in Android Apps2024
- Test Transfer Across Mobile Apps Through Semantic Mapping2019
- Test Migration Between Mobile Apps with Similar Functionality2019
- Humanoid: A Deep Learning-Based Approach to Automated Black-box Android App Testing2019
- Analyses for specific defects in android applications: a surveyFrontiers of Computer Science, 2019
- Static window transition graphs for AndroidAutomated Software Engineering, 2018
- Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generationSoftware: Practice and Experience, 2018
- Automated cross-platform inconsistency detection for mobile apps2017
- An Empirical Analysis of Bug Reports and Bug Fixing in Open Source Android Apps2013
- A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications2013