Non-uniform Timestep Sampling: Towards Faster Diffusion Model Training
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 7036-7045
Abstract
Diffusion models have garnered significant success in generative tasks, emerging as the predominant model in this domain. Despite their success, the substantial computational resources required for training diffusion models restrict their practical applications. In this paper, we resort to the optimal transport theory to accelerate the training of diffusion models, providing an in-depth analysis of the forward diffusion process. It shows that the upper bound on the Wasserstein distance of the distribution between any two timesteps in the diffusion process is an exponential decrease of the initial distance by a factor of times. This finding suggests that the state distribution of the diffusion model has a non-uniform rate of change at different points in time, thus highlighting the different importance of the diffusion timestep. To this end, we propose a novel non-uniform timestep sampling method based on the Bernoulli distribution, which favors more frequent sampling in significant timestep intervals. The key idea is to make the model focus on timesteps with larger differences, thus accelerating the training of the diffusion model. Experiments on benchmark datasets reveal that the proposed method significantly reduces the computational overhead while improving the quality of the generated images.Keywords
Funding Information
- Science and Technology Commission of Shanghai Municipality (22DZ2229005)
- National Natural Science Foundation of China (61927809)
This publication has 12 references indexed in Scilit:
- Rethinking Open-World DeepFake Attribution with Multi-perspective Sensory LearningInternational Journal of Computer Vision, 2024
- CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-Spoofing2024
- Source-Free Domain Adaptation With Domain Generalized Pretraining for Face Anti-SpoofingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- DPM-OT: A New Diffusion Probabilistic Model Based on Optimal Transport2023
- Contrastive Pseudo Learning for Open-World DeepFake Attribution2023
- Re-thinking Data Availablity Attacks Against Deep Neural NetworksarXiv, 2023
- Entropy Dissipation for Degenerate Stochastic Differential Equations via Sub-Riemannian Density ManifoldEntropy, 2023
- Label-Free Regional Consistency for Image-to-Image Translation2021
- A Style-Based Generator Architecture for Generative Adversarial NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Optimal Transport2009