All are Worth Words: A ViT Backbone for Diffusion Models
Preprint
Abstract
Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and class-conditional image generation, as well as text-to-image generation tasks, where U-ViT is comparable if not superior to a CNN-based U-Net of a similar size. In particular, a latent diffusion model with a small U-ViT achieves a record-breaking FID of 5.48 in text-to-image generation on MS-COCO, among methods without accessing large external datasets during the training of generative models. Besides, our results suggest that, for diffusion-based image modeling, the long skip connection is crucial while the down-sampling and up-sampling operators in CNN-based U-Net are not always necessary. We believe that U-ViT can provide insights for future research on backbones in diffusion models and benefit generative modeling on large scale cross-modality datasets.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Frido: Feature Pyramid Diffusion for Complex Scene Image SynthesisProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- Towards Language-Free Training for Text-to-Image Generation2022
- ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models2021
- Microsoft COCO: Common Objects in Context2014