Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Caching
Authors:
- Zhixin ZhengZZZhixin Zheng
- Xinyu WangXWXinyu Wang
- Chang ZouCZChang Zou
- Shaobo WangSWShaobo Wang
- Linfeng ZhangLZLinfeng Zhang
Preprint
Abstract
Diffusion transformers have gained significant attention in recent years for their ability to generate high-quality images and videos, yet still suffer from a huge computational cost due to their iterative denoising process. Recently, feature caching has been introduced to accelerate diffusion transformers by caching the feature computation in previous timesteps and reusing it in the following timesteps, which leverage the temporal similarity of diffusion models while ignoring the similarity in the spatial dimension. In this paper, we introduce Cluster-Driven Feature Caching (ClusCa) as an orthogonal and complementary perspective for previous feature caching. Specifically, ClusCa performs spatial clustering on tokens in each timestep, computes only one token in each cluster and propagates their information to all the other tokens, which is able to reduce the number of tokens by over 90%. Extensive experiments on DiT, FLUX and HunyuanVideo demonstrate its effectiveness in both text-to-image and text-to-video generation. Besides, it can be directly applied to any diffusion transformer without requirements for training. For instance, ClusCa achieves 4.96x acceleration on FLUX with an ImageReward of 99.49%, surpassing the original model by 0.51%. The code is available at https://github.com/Shenyi-Z/Cache4Diffusion.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic ModelsMachine Intelligence Research, 2025
- DiTFastAttn: Attention Compression for Diffusion Transformer Models2024
- DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization2024
- Structural Pruning for Diffusion Models2023
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation2023
- DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2022
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness2022
- Photorealistic Text-To-Image Diffusion Models with Deep Language Understanding2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015