InterTrack: Tracking Human Object Interaction without Object Templates
Authors:
Preprint
Abstract
Tracking human object interaction from videos is important to understand human behavior from the rapidly growing stream of video data. Previous video-based methods require predefined object templates while single-image-based methods are template-free but lack temporal consistency. In this paper, we present a method to track human object interaction without any object shape templates. We decompose the 4D tracking problem into per-frame pose tracking and canonical shape optimization. We first apply a single-view reconstruction method to obtain temporally-inconsistent per-frame interaction reconstructions. Then, for the human, we propose an efficient autoencoder to predict SMPL vertices directly from the per-frame reconstructions, introducing temporally consistent correspondence. For the object, we introduce a pose estimator that leverages temporal information to predict smooth object rotations under occlusions. To train our model, we propose a method to generate synthetic interaction videos and synthesize in total 10 hour videos of 8.5k sequences with full 3D ground truth. Experiments on BEHAVE and InterCap show that our method significantly outperforms previous template-based video tracking and single-frame reconstruction methods. Our proposed synthetic video dataset also allows training video-based methods that generalize to real-world videos. Our code and dataset will be publicly released.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream2023
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding2022
- BANMo: Building Animatable 3D Neural Models from Many Casual Videos2022
- OnePose: One-Shot Object Pose Estimation without CAD Models2022
- NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions2022
- LASR: Learning Articulated Shape Reconstruction from a Monocular Video2021
- Holistic 3D Human and Scene Mesh Estimation from Single View Images2021
- Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction2020
- Learning Joint Reconstruction of Hands and Manipulated Objects2019
- 3D-CODED: 3D Correspondences by Deep Deformation2018