image2emmet: Automatic code generation from web user interface image
- 28 June 2021
- journal article
- research article
- Published by Wiley in Journal of Software: Evolution and Process
- Vol. 33 (8) , e2369
Abstract
Web development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front‐end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back‐end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time‐consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML‐CSS) dataset in an automatic and efficient way. The HTML‐CSS dataset used for model training consists of HTML‐CSS code and its display images. Second, the faster region‐based convolutional neural network (CNN) (R‐CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short‐term memory (LSTM) to transform the UI component into the HTML‐CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML‐CSS code.Keywords
Funding Information
- National Natural Science Foundation of China (61872312, 61972335, 62002309)
- Six Talent Peaks Project in Jiangsu Province (RJFW‐053)
- Project 333 of Jiangsu Province
This publication has 30 references indexed in Scilit:
- SSD: Single Shot MultiBox Detector2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Fast R-CNN2015
- Reverse Engineering Mobile Application User Interfaces with REMAUI (T)2015
- Fully convolutional networks for semantic segmentation2015
- Generating Java Code from UML Class and Sequence Diagrams2011
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Long Short-Term MemoryNeural Computation, 1997
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994
- A Computational Approach to Edge DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 1986