Motion Modes: What Could Happen Next?
Authors:
Preprint
Abstract
Predicting diverse object motions from a single static image remains challenging, as current video generation models often entangle object movement with camera motion and other scene changes. While recent methods can predict specific motions from motion arrow input, they rely on synthetic data and predefined motions, limiting their application to complex scenes. We introduce Motion Modes, a training-free approach that explores a pre-trained image-to-video generator's latent distribution to discover various distinct and plausible motions focused on selected objects in static images. We achieve this by employing a flow generator guided by energy functions designed to disentangle object and camera motion. Additionally, we use an energy inspired by particle guidance to diversify the generated motions, without requiring explicit training data. Experimental results demonstrate that Motion Modes generates realistic and varied object animations, surpassing previous methods and even human predictions regarding plausibility and diversity. Project Webpage: https://motionmodes.github.io/Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model2024
- DiffUHaul: A Training-Free Method for Object Dragging in Images2024
- DragAnything: Motion Control for Anything Using Entity Representation2024
- Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D2024
- DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing2024
- Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images2023
- Learning to predict part mobility from a single static snapshotACM Transactions on Graphics, 2017