DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation
Top Cited Papers
Open Access
- 19 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (FSE) , 219-241
Abstract
Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic. To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI’s generalizability through successful applications to Flutter and ArkUI frameworks. User studies with professional developers confirm that DeclarUI’s generated code meets industrial-grade standards in code availability, modification time, readability, and maintainability. By streamlining app development, improving efficiency, and fostering designer-developer collaboration, DeclarUI offers a practical solution to the persistent challenges in mobile UI development.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Deep learning models for digital image processing: a reviewArtificial Intelligence Review, 2024
- Exploring Distributional Shifts in Large Language Models for Code AnalysisarXiv, 2023
- A Multimodal Deep Learning Model Using Text, Image, and Code Data for Improving Issue Classification TasksApplied Sciences, 2023
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionarXiv, 2023
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation ModelsarXiv, 2023
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingarXiv, 2022
- On the Transferability of Pre-trained Language Models for Low-Resource Programming LanguagesarXiv, 2022
- Code generation from a graphical user interface via attention-based encoder–decoder modelMultimedia Systems, 2021
- Recent Progress in Automated Code Generation from GUI Images Using Machine Learning TechniquesJUCS - Journal of Universal Computer Science, 2020
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004