Dreamix: Video Diffusion Models are General Video Editors
Authors:
Preprint
Abstract
Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtaining high-fidelity to the original video requires retaining some of its high-resolution information, we add a preliminary stage of finetuning the model on the original video, significantly boosting fidelity. We propose to improve motion editability by a new, mixed objective that jointly finetunes with full temporal attention and with temporal attention masking. We further introduce a new framework for image animation. We first transform the image into a coarse video by simple image processing operations such as replication and perspective geometric projections, and then use our general video editor to animate it. As a further application, we can use our method for subject-driven video generation. Extensive qualitative and numerical experiments showcase the remarkable editing ability of our method and establish its superior performance compared to baseline methods.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation2023
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- SpaText: Spatio-Textual Representation for Controllable Image Generation2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- MAGVIT: Masked Generative Video Transformer2023
- InstructPix2Pix: Learning to Follow Image Editing Instructions2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- Image Super-Resolution Via Iterative RefinementIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- A Connection Between Score Matching and Denoising AutoencodersNeural Computation, 2011