ReVideo: Remake a Video with Motion and Content Control
Authors:
Preprint
Abstract
Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Video Editing via Factorized Diffusion Distillation2024
- CoTracker: It Is Better to Track Together2024
- MotionDirector: Motion Customization of Text-to-Video Diffusion Models2024
- DynamiCrafter: Animating Open-Domain Images with Video Diffusion Priors2024
- DragAnything: Motion Control for Anything Using Entity Representation2024
- RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models2024
- Peekaboo: Interactive Video Generation via Masked-Diffusion2024
- Dream Video: Composing Your Dream Videos with Customized Subject and Motion2024
- Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis2024
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis2024