Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12420-12427
Abstract
Dense visual correspondence plays a vital role in robotic perception. This work focuses on establishing the dense correspondence between a pair of images that captures dynamic scenes undergoing substantial transformations. We introduce Doduo to learn general dense visual correspondence from in-the-wild images and videos without ground truth supervision. Given a pair of images, it estimates the dense flow field encoding the displacement of each pixel in one image to its corresponding pixel in the other image. Doduo uses flow-based warping to acquire supervisory signals for the training. Incorporating semantic priors with self-supervised flow training, Doduo produces accurate dense correspondence robust to the dynamic changes of the scenes. Trained on an in-the-wild video dataset, Doduo illustrates superior performance on point-level correspondence estimation over existing self-supervised correspondence learning baselines. We also apply Doduo to articulation estimation and zero-shot goal-conditioned manipulation, underlining its practical applications in robotics. Code and additional visualizations are available at https://ut-austin-rpl.github.io/Doduo/Keywords
All Related Versions
This publication has 51 references indexed in Scilit:
- Semantic-Aware Fine-Grained Correspondence2022
- Learning Feature Descriptors Using Camera Pose Supervision2020
- What Matters in Unsupervised Optical Flow2020
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow2020
- Unsupervised Learning of Multi-Frame Optical Flow with Occlusions2018
- YouTube-VOS: Sequence-to-Sequence Video Object Segmentation2018
- Tracking Emerges by Colorizing Videos2018
- Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness2016
- LIFT: Learned Invariant Feature Transform2016
- Learning Image Matching by Simply Watching Video2016