A novel code generator for graphical user interfaces
Authors:
Open Access
- 21 November 2023
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 13 (1) , 1-16
Abstract
Graphical user interfaces (GUIs) are widely used in human–computer interaction, providing a convenient interface for operation. Automating the conversion of GUI design images into source code can significantly reduce the coding workload for front-end developers. Detecting elements in GUI images is a key challenge in achieving automatic GUI code generation and is crucial for tasks such as GUI automation and testing. However, current state-of-the-art methods do not fully consider the unique characteristics of GUI images and elements, and they lack the required high localization accuracy, resulting in low detection accuracy for GUI element boxes. In this paper, we propose GUICG, an automatic GUI code generator that combines deep neural networks with image processing techniques to efficiently detect GUI elements from GUI images and generate front-end code. We empirically investigate various deep learning approaches and image processing methods for GUI component detection. Based on a comprehensive understanding of their performance and characteristics, we design GUICG by fusing image processing with a deep learning-based target detection model, achieving state-of-the-art performance. GUICG outperforms existing methods in accuracy and F1 score for component detection tasks, while producing human-readable code with a logical structure. Furthermore, we conduct an ablation study to quantitatively assess the impact of each key element in GUICG.Keywords
Funding Information
- National Natural Science Foundation of China (61971316, 61971316, 61971316)
This publication has 43 references indexed in Scilit:
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Fast R-CNN2015
- Reverse Engineering Mobile Application User Interfaces with REMAUI (T)2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- An Overview of the Tesseract OCR EngineNinth International Conference on Document Analysis and Recognition (ICDAR 2007), 2007
- Efficient component labeling of images of arbitrary dimension represented by linear bintreesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1988
- A Computational Approach to Edge Detection1987
- A Computational Approach to Edge DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 1986
- Topological structural analysis of digitized binary images by border followingComputer Vision, Graphics, and Image Processing, 1985
- An iterative procedure for the polygonal approximation of plane curvesComputer Graphics and Image Processing, 1972