DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
Authors:
Preprint
Abstract
Enhancing a still image with motion offers more engaged visual experience. Traditional image animation techniques mainly focus on animating natural scenes with random dynamics, such as clouds and fluid, and thus limits their applicability to generic visual contents. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich image embedding space using a learnable image encoding network, which facilitates the video model to digest the image content compatibly. However, some visual details still struggle to be preserved in the resulting videos. To supplement more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results reveal that our proposed method produces visually convincing animated videos, exhibiting both natural motions and high fidelity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors. The source code will be released upon publication.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion ModelsInternational Journal of Computer Vision, 2024
- Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video GenerationInternational Journal of Computer Vision, 2024
- Stochastic Image-to-Video Synthesis using cINNs2021
- DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image2020
- Analyzing and Improving the Image Quality of StyleGAN2020
- Time Flies: Animating a Still Image With Time-Lapse Video As Reference2020
- Photo Wake-Up: 3D Character Animation From a Single Photo2019
- Animating Arbitrary Objects via Deep Motion Transfer2019
- Flow-Grounded Spatial-Temporal Video Prediction from Still Images2018
- Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks2018