VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Control
Authors:
Preprint
Abstract
Despite significant advancements in video generation, inserting a given object into videos remains a challenging task. The difficulty lies in preserving the appearance details of the reference object and accurately modeling coherent motions at the same time. In this paper, we propose VideoAnydoor, a zero-shot video object insertion framework with high-fidelity detail preservation and precise motion control. Starting from a text-to-video model, we utilize an ID extractor to inject the global identity and leverage a box sequence to control the overall motion. To preserve the detailed appearance and meanwhile support fine-grained motion control, we design a pixel warper. It takes the reference image with arbitrary key-points and the corresponding key-point trajectories as inputs. It warps the pixel details according to the trajectories and fuses the warped features with the diffusion U-Net, thus improving detail preservation and supporting users in manipulating the motion trajectories. In addition, we propose a training strategy involving both videos and static images with a reweight reconstruction loss to enhance insertion quality. VideoAnydoor demonstrates significant superiority over existing methods and naturally supports various downstream applications (e.g., talking head generation, video virtual try-on, multi-region editing) without task-specific fine-tuning.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- CoTracker: It Is Better to Track Together2024
- AnyDoor: Zero-shot Object-level Image Customization2024
- VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models2024
- MVImgNet: A Large-scale Dataset of Multi-view Images2023
- Paint by Example: Exemplar-based Image Editing with Diffusion Models2023
- High-Fidelity Guided Image Synthesis with Latent Diffusion Models2023
- Multi-Concept Customization of Text-to-Image Diffusion2023
- Large-scale Video Panoptic Segmentation in the Wild: A Benchmark2022
- Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset2021
- VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild2021