SMF: Template-free and Rig-free Animation Transfer using Kinetic Codes
Authors:
- Niladri Shekhar DuttNDNiladri Shekhar Dutt
Preprint
Abstract
Animation retargeting involves applying a sparse motion description (e.g., 2D/3D keypoint sequences) to a given character mesh to produce a semantically plausible and temporally coherent full-body motion. Existing approaches come with a mix of restrictions - they require annotated training data, assume access to template-based shape priors or artist-designed deformation rigs, suffer from limited generalization to unseen motion and/or shapes, or exhibit motion jitter. We propose Self-supervised Motion Fields (SMF) as a self-supervised framework that can be robustly trained with sparse motion representations, without requiring dataset specific annotations, templates, or rigs. At the heart of our method are Kinetic Codes, a novel autoencoder-based sparse motion encoding, that exposes a semantically rich latent space simplifying large-scale training. Our architecture comprises dedicated spatial and temporal gradient predictors, which are trained end-to-end. The resultant network, regularized by the Kinetic Codes's latent space, has good generalization across shapes and motions. We evaluated our method on unseen motion sampled from AMASS, D4D, Mixamo, and raw monocular video for animation transfer on various characters with varying shapes and topology. We report a new SoTA on the AMASS dataset in the context of generalization to unseen motion. Project webpage at https://motionfields.github.io/Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- TRAM: Global Trajectory and Motion of 3D Humans from in-the-Wild Videos2024
- SMPL: A Skinned Multi-Person Linear Model2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- 4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface2021
- EMU: Efficient Muscle Simulation in Deformation SpaceComputer Graphics Forum, 2020
- Deep High-Resolution Representation Learning for Human Pose Estimation2019
- Phase-functioned neural networks for character controlACM Transactions on Graphics, 2017
- Recurrent Network Models for Human Dynamics2015
- Active volumetric musculoskeletal systemsACM Transactions on Graphics, 2014
- Deformation transfer for triangle meshesACM Transactions on Graphics, 2004