TextDiffuser: Diffusion Models as Text Painters
Preprint
Abstract
Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at \url{https://aka.ms/textdiffuser}.Keywords
All Related Versions
This publication has 53 references indexed in Scilit:
- Diff-Font: Diffusion Model for Robust One-Shot Font GenerationInternational Journal of Computer Vision, 2024
- OCR-RTPS: an OCR-based real-time positioning system for the valet parkingApplied Intelligence, 2023
- Scene Text Recognition with Permuted Autoregressive Sequence Models2022
- Detecting Tampered Scene Text in the Wild2022
- Mask-guided GAN for robust text editing in the sceneNeurocomputing, 2021
- Scene Text Image Super-Resolution in the Wild2020
- Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations2020
- Synthetic Data for Text Localisation in Natural Images2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Filling-in by joint interpolation of vector fields and gray levelsIEEE Transactions on Image Processing, 2001