Bridging Design and Development with Automated Declarative UI Code Generation
Preprint
Abstract
Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic. To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI's generalizability through successful applications to Flutter and ArkUI frameworks.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- Prototype2Code: End-to-End Front-End Code Generation From UI Design Prototypes2024
- Deep learning models for digital image processing: a reviewArtificial Intelligence Review, 2024
- A novel code generator for graphical user interfacesScientific Reports, 2023
- Statistical Inferences for Complex Dependence of Multimodal Imaging DataJournal of the American Statistical Association, 2023
- Knowledge Graphs: Opportunities and ChallengesArtificial Intelligence Review, 2023
- Assemble Foundation Models for Automatic Code Summarization2022
- Code generation from a graphical user interface via attention-based encoder–decoder modelMultimedia Systems, 2021
- DroidBot: a lightweight UI-Guided test input generator for android2017
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004