Make Pixels Dance: High-Dynamic Video Generation
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8850-8860
Abstract
Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortu-nately, current state-of-the-art video generation methods, primarily focusing on text-to- video generation, tend to pro-duce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text in-structions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Structure and Content-Guided Video Synthesis with Diffusion Models2023
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation2023
- Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models2023
- Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models2023
- Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval2021
- Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GANInternational Journal of Computer Vision, 2020
- To Create What You Tell2017
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016