Muse: Text-To-Image Generation via Masked Generative Transformers
Authors:
- Huiwen ChangHCHuiwen Chang
- Han ZhangHZHan Zhang
- Jarred BarberJBJarred Barber
- Aj MaschinotAMAj Maschinot
- Jose LezamaJLJose Lezama
- Lu JiangLJLu Jiang
- Ming-Hsuan YangMYMing-Hsuan Yang
- Kevin MurphyKMKevin Murphy
- William T. FreemanWFWilliam T. Freeman
- Michael RubinsteinMRMichael Rubinstein
- Yuanzhen LiYLYuanzhen Li
- Dilip KrishnanDKDilip Krishnan
Preprint
Abstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.ioKeywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting2023
- MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- Towards Language-Free Training for Text-to-Image Generation2022
- DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis2022
- Masked Autoencoders Are Scalable Vision Learners2022
- Data and its (dis)contents: A survey of dataset development and use in machine learning researchPatterns, 2021
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018