Character-Aware Models Improve Visual Text Rendering
Preprint
Abstract
Current image generation models struggle to reliably produce well-formed visual text. In this paper, we investigate a key contributing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word's visual makeup as a series of glyphs. To quantify this effect, we conduct a series of experiments comparing character-aware vs. character-blind text encoders. In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell). Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark). Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte ModelsTransactions of the Association for Computational Linguistics, 2022
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language RepresentationTransactions of the Association for Computational Linguistics, 2022
- Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens2022
- What do tokens know about their characters and how do they know it?2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Deep Contextualized Word Representations2018
- Neural Machine Translation of Rare Words with Subword Units2016