Truncate Diffusion: Efficient Video Editing With Low-Rank Truncate
- 4 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 7093-7108
Abstract
Diffusion models have demonstrated remarkable capabilities for text-to-video (T2V) editing tasks, relying on fine-tuning for pretrained text-to-image (T2I) diffusion models with only one video-prompt pair. However, conventional fine-tuning approaches require tuning and storing numerous parameters for each video, leading to substantial parameter and memory costs. To mitigate these issues, we propose Truncating Diffusion, an efficient fine-tuning method for video editing that optimizes both parameter and memory usage. Specifically, we propose the Truncating Diffusion module, which is designed with a focus on module architecture and initialization, specifically targeting optimization with a small training set, such as a single video-prompt pair. Theoretical analysis using the Johnson-Lindenstrauss lemma and the Eckart-Young-Mirsky theorem shows that Truncating Diffusion can achieve a minimal Frobenius norm distance to the original attention algorithm with appropriate initialization, which enhances the ease of optimization and improves video editing performance. During fine-tuning, the Truncating Diffusion module integrates seamlessly with the original diffusion model. We freeze the weights of the denoising network within the original pretrained diffusion model, updating only the introduced low-rank alternatives to ensure parameter and memory efficiency. Additionally, we propose Latent Flow Loss and Bidirectional Inter-Frame Attention (BIFA) to improve temporal consistency in synthesized videos. The Latent Flow Loss leverages global temporal information from the input video during training, while BIFA utilizes local temporal information from adjacent frames during inference. These enhancements do not incur additional memory or parameter costs during fine-tuning. Comparisons with state-of-the-art approaches demonstrate that Truncating Diffusion provides superior text alignment, video quality, and inter-frame temporal consistency in video editing. Importantly, Truncating Diffusion requires fine-tuning only 3.2% of the parameters and uses just 62% of the memory compared to the baseline model.Keywords
Funding Information
- National Natural Science Foundation of China (U2336212, 62272411)
- Key R&D Projects in Zhejiang Province (2024C01106, 2024C01102)
- Zhejiang NSF (LRG25F020001)
- National Key R&D Program of China (2018AAA0101900)
- Huawei Cloud (TC20231124047-2024-01)
This publication has 27 references indexed in Scilit:
- KronA: Parameter-Efficient Tuning with Kronecker Adapter2025
- SimDA: Simple Diffusion Adapter for Efficient Video Generation2024
- VidToMe: Video Token Merging for Zero-Shot Video Editing2024
- Video-P2P: Video Editing with Cross-Attention Control2024
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Extensions of Lipschitz mappings into a Hilbert space1984
- The Approximation of One Matrix by Another of Lower RankPsychometrika, 1936