LDT: Efficient Scalable Video Generation Using Linear Diffusion Transformer
- 27 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-12
Abstract
Diffusion-based models have established new bench marks in high-quality video generation. However, they face significant challenges regarding computational efficiency, particularly when dealing with long sequences, because of their inherent quadratic complexity. To address these challenges, we introduce the Linear Diffusion Transformer (LDT), which utilizes the linear attention mechanism to reduce computational complexity to a linear scale. This advancement facilitates the efficient generation of longer video sequences without sacrificing the quality of the output. Among the key innovations in our approach include two sophisticated positional encoding techniques: Exponential Relative Positional Encoding (ERPE) and 3D Linearized Relative Positional Encoding (3D-LRPE). By capturing local correlations between video tokens through these position-aware encodings, our model effectively preserves computational efficiency while maintaining the ability to model fine-grained spatiotemporal relationships. These techniques enhance the capability of the model to capture intricate spatial and temporal dependencies effectively. Furthermore, we propose the Spatial-Global Transformer (SGT), which alternates between spatial attention and global sequence attention to improve video modeling efficacy. Extensive experimental evaluations demonstrate that LDT not only achieves state of-the-art efficiency, but also excels in generation performance. In particular, in the “train short, test long” setting, our method exhibits remarkable scalability as the length of the video frames increases, underscoring its robustness for longer video sequences.Keywords
Funding Information
- National Natural Science Foundation of China (62525115)
- Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2024014)
This publication has 36 references indexed in Scilit:
- Swap Attention in Spatiotemporal Diffusions for Text-to-Video GenerationInternational Journal of Computer Vision, 2025
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion ModelsInternational Journal of Computer Vision, 2024
- Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video GenerationInternational Journal of Computer Vision, 2024
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN22022
- Video Swin Transformer2022
- Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks2018
- MoCoGAN: Decomposing Motion and Content for Video Generation2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015