Track2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot Manipulation
Preprint
Abstract
We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation: interacting with unseen objects in novel scenes without test-time adaptation. While typical approaches rely on a large amount of demonstration data for such generalization, we propose an approach that leverages web videos to predict plausible interaction plans and learns a task-agnostic transformation to obtain robot actions in the real world. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal, and can be trained with diverse videos on the web including those of humans and robots manipulating everyday objects. We use these 2D track predictions to infer a sequence of rigid transforms of the object to be manipulated, and obtain robot end-effector poses that can be executed in an open-loop manner. We then refine this open-loop plan by predicting residual actions through a closed loop policy trained with a few embodiment-specific demonstrations. We show that this approach of combining scalably learned track prediction with a residual policy requiring minimal in-domain robot-specific data enables zero-shot robot manipulation, and present a wide array of real-world robot manipulation results across unseen tasks, objects, and scenes. https://homangab.github.io/track2act/Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Photorealistic Video Generation with Diffusion Models2024
- CoTracker: It Is Better to Track Together2024
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos2022
- Segmentation-Driven 6D Object Pose Estimation2019
- Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering2019
- Hand Pose Estimation via Latent 2.5D Heatmap Regression2018
- In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video2018
- Cross-Modal Deep Variational Hand Pose Estimation2018
- A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching2013
- SIFT Flow: Dense Correspondence across Different Scenes2008