LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer
Authors:
- Ning YuNYNing Yu
- Zeyuan ChenZCZeyuan Chen
- Rui MengRMRui Meng
- Gang WuGWGang Wu
- Paul JoselPJPaul Josel
- Juan Carlos NieblesJNJuan Carlos Niebles
- Caiming XiongCXCaiming Xiong
- Ran XuRXRan Xu
Preprint
Abstract
Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs are skill-demanding, time-consuming, and non-scalable to batch production. Although generative models emerge to make design automation no longer utopian, it remains non-trivial to customize designs that comply with designers' multimodal desires, i.e., constrained by background images and driven by foreground contents. In this study, we propose \textit{LayoutDETR} that inherits the high quality and realism from generative modeling, in the meanwhile reformulating content-aware requirements as a detection problem: we learn to detect in a background image the reasonable locations, scales, and spatial relations for multimodal elements in a layout. Experiments validate that our solution yields new state-of-the-art performance for layout generation on public benchmarks and on our newly-curated ads banner dataset. For practical usage, we build our solution into a graphical system that facilitates user studies. We demonstrate that our designs attract more subjective preferences than baselines by significant margins. Our code, models, dataset, graphical system, and demos are available at https://github.com/salesforce/LayoutDETR.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- BLT: Bidirectional Layout Transformer for Controllable Layout Generation2022
- Neural Design Network: Graphic Layout Generation with Constraints2020
- Inclusive GAN: Improving Data and Minority Coverage in Generative Models2020
- Understanding Programmatic Creative: The Role of AIJournal of Advertising, 2019
- Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression2019
- Compositing-Aware Image Search2018
- Non-local Neural Networks2018
- YOLO9000: Better, Faster, Stronger2017
- Automatic Understanding of Image and Video Advertisements2017
- You Only Look Once: Unified, Real-Time Object Detection2016