DyDiT++: Diffusion Transformers With Timestep and Spatial Dynamics for Efficient Visual Generation
- 15 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (5) , 5755-5773
Abstract
Diffusion Transformer (DiT), an emerging diffusion model for visual generation, has demonstrated superior performance but suffers from substantial computational costs. Our investigations reveal that these costs primarily stem from the static inference paradigm, which inevitably introduces redundant computation in certain diffusion timesteps and spatial regions. To overcome this inefficiency, we propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions. Specifically, we introduce a Timestep-wise Dynamic Width (TDW) approach that adapts model width conditioned on the generation timesteps. In addition, we design a Spatial-wise Dynamic Token (SDT) strategy to avoid redundant computation at unnecessary spatial locations. TDW and SDT can be seamlessly integrated into DiT and significantly accelerate the generation process. Building on these designs, we present an extended version, DyDiT++, with improvements in three key aspects. First, it extends the generation mechanism of DyDiT beyond diffusion to flow matching, demonstrating that our method can also accelerate flow-matching-based generation, enhancing its versatility. Furthermore, we enhance DyDiT to tackle more complex visual generation tasks, including video generation and text-to-image generation, thereby broadening its real-world applications. Finally, to address the high cost of full fine-tuning and democratize technology access, we investigate the feasibility of training DyDiT in a parameter-efficient manner and introduce timestep-based dynamic LoRA (TD-LoRA). Extensive experiments on diverse visual generation models, including DiT, SiT, Latte, and FLUX, demonstrate the effectiveness of DyDiT++. Remarkably, with $< $ 3% additional fine-tuning iterations, our approach reduces the FLOPs of DiT-XL by 51%, yielding 1.73× realistic speedup on hardware, and achieves a competitive FID score of 2.07 on ImageNet.
Keywords
All Related Versions
Funding Information
- Damo Academy through Damo Academy Research Intern Program
- NUS startup (MOE Tier-1)
- NUS startup (ByteDance)
- NUS startup (ARCTIC)
- Society for Musicology in Ireland (A8001104-00-00)
- Alibaba
This publication has 43 references indexed in Scilit:
- DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic ModelsMachine Intelligence Research, 2025
- DiffiT: Diffusion Vision Transformers for Image Generation2024
- SiT: Exploring Flow and Diffusion-Based Generative Models with Scalable Interpolant Transformers2024
- Learning to Weight Samples for Dynamic Early-Exiting Networks2022
- Channel Selection Using Gumbel Softmax2020
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Food-101 – Mining Discriminative Components with Random Forests2014
- ImageNet: A large-scale hierarchical image database2009