Accelerating Diffusion Sampling with Classifier-based Feature Distillation
Open Access
- 1 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 810-815
Abstract
Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of N-step teacher sampler with N/2-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed Classifier-based Feature Distillation (CFD). Instead of aligning output images, we distill teacher’s sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code is available at https://github.com/zju-SWJ/RCFD.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Zhejiang University
This publication has 10 references indexed in Scilit:
- When Object Detection Meets Knowledge Distillation: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Cross-Image Relational Knowledge Distillation for Semantic Segmentation2022
- Knowledge Distillation with the Reused Teacher Classifier2022
- Channel-wise Knowledge Distillation for Dense Prediction2021
- Cross-Layer Distillation with Semantic CalibrationProceedings of the AAAI Conference on Artificial Intelligence, 2021
- Generative Adversarial Networks (GANs)ACM Computing Surveys, 2021
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Online Knowledge Distillation with Diverse PeersProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016