Adding Conditional Control to Text-to-Image Diffusion Models
Preprint
Abstract
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- OneFormer: One Transformer to Rule Universal Image Segmentation2023
- MaskSketch: Unpaired Structure-guided Masked Image Generation2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks2022
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Learning representations by back-propagating errorsNature, 1986