Q-Diffusion: Quantizing Diffusion Models
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17489-17499
Abstract
Diffusion models have achieved great success in image synthesis through iterative noise estimation using deep neural networks. However, the slow inference, high memory consumption, and computation intensity of the noise estimation model hinder the efficient adoption of diffusion models. Although post-training quantization (PTQ) is considered a go-to compression method for other tasks, it does not work out-of-the-box on diffusion models. We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture of the diffusion models, which compresses the noise estimation network to accelerate the generation process. We identify the key difficulty of diffusion model quantization as the changing output distributions of noise estimation networks over multiple time steps and the bimodal activation distribution of the shortcut layers within the noise estimation network. We tackle these challenges with timestep-aware calibration and split shortcut quantization in this work. Experimental results show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance (small FID change of at most 2.34 compared to >100 for traditional PTQ) in a training-free manner. Our approach can also be applied to text-guided image generation, where we can run stable diffusion in 4-bit weights with high generation quality for the first time.Keywords
All Related Versions
Funding Information
- Intel Corporation
This publication has 16 references indexed in Scilit:
- On Distillation of Guided Diffusion Models2023
- NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers2023
- Post-Training Quantization on Diffusion Models2023
- Unsupervised Selective Labeling for More Effective Semi-supervised Learning2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- SRDiff: Single image super-resolution with diffusion probabilistic modelsNeurocomputing, 2022
- Learning Gradient Fields for Shape Generation2020
- ZeroQ: A Novel Zero Shot Quantization Framework2020
- Analyzing and Improving the Image Quality of StyleGAN2020
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015