4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface
Authors:
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12686-12696
Abstract
Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics. However, due to occlusions and physical limitations of range sensors, existing methods only handle the visible surface, thus causing discontinuities and in-completeness in the motion field. To this end, we introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. 4DComplete takes as input a partial shape and motion observation, extracts 4D time-space embedding, and jointly infers the missing geometry and motion field using a sparse fully-convolutional network. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D, which consists of 1,972 animation sequences spanning 31 different animals or humanoid categories with dense 4D annotation. Experiments show that 4DComplete 1) reconstructs high-resolution volumetric shape and motion field from a partial observation, 2) learns an entangled 4D feature representation that benefits both shape and motion estimation, 3) yields more accurate and natural deformation than classic non-rigid priors such as As-Rigid-As-Possible (ARAP) deformation, and 4) generalizes well to unseen objects in real-world sequences.Keywords
This publication has 45 references indexed in Scilit:
- DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time2015
- Real-time non-rigid reconstruction using an RGB-D cameraACM Transactions on Graphics, 2014
- FAUST: Dataset and Evaluation for 3D Mesh Registration2014
- Performance Capture of Interacting Characters with Handheld Kinects2012
- Articulated mesh animation from multi-view silhouettes2008
- Global Correspondence Optimization for Non‐Rigid Registration of Depth ScansComputer Graphics Forum, 2008
- Embedded deformation for shape manipulationACM Transactions on Graphics, 2007
- SCAPE2005
- A volumetric method for building complex models from range images1996
- The Efficient Generation of Random Orthogonal Matrices with an Application to Condition EstimatorsSIAM Journal on Numerical Analysis, 1980