DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Preprint
Abstract
Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- A Multimodal LLM for Chart Understanding and Generation2025
- Improved Baselines with Visual Instruction Tuning2024
- ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural LanguageIEEE Transactions on Visualization and Computer Graphics, 2024
- VINS: Visual Search for Mobile User Interface Design2021
- Brain tumor segmentation using K‐means clustering and deep learning with synthetic data augmentation for classificationMicroscopy Research and Technique, 2021
- Object detection for graphical user interface: old fashioned or deep learning or a combination?2020
- Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning2019
- Automated Cross-Platform GUI Code Generation for Mobile Apps2019
- Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile AppsIEEE Transactions on Software Engineering, 2018
- From UI design image to GUI skeleton2018