The Functional Correspondence Problem
Authors:
Open Access
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Proceedings of SPIE
- p. 15752-15761
Abstract
The ability to find correspondences in visual data is the essence of most computer vision tasks. But what are the right correspondences? The task of visual correspondence is well defined for two different images of same object instance. In case of two images of objects belonging to same category, visual correspondence is reasonably well-defined in most cases. But what about correspondence between two objects of completely different category – e.g., a shoe and a bottle? Does there exist any correspondence? Inspired by humans’ ability to: (a) generalize beyond semantic categories and; (b) infer functional affordances, we introduce the problem of functional correspondences in this paper. Given images of two objects, we ask a simple question: what is the set of correspondences between these two images for a given task? For example, what are the correspondences between a bottle and shoe for the task of pounding or the task of pouring. We introduce a new dataset: FunKPoint that has ground truth correspondences for 10 tasks and 20 object categories. We also introduce a modular task-driven representation for attacking this problem and demonstrate that our learned representation is effective for this task. But most importantly, because our supervision signal is not bound by semantics, we show that our learned representation can generalize better on few-shot classification problem. We hope this paper will inspire our community to think beyond semantics and focus more on cross-category generalization and learning representations for robotics tasks.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Learning to push by grasping: Using multiple tasks for effective learning2017
- Deep Residual Learning for Image Recognition2016
- People Watching: Human Actions as a Cue for Single View Geometry2012
- Shape Matching and Object Recognition Using Low Distortion Correspondences2005
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Object recognition from local scale-invariant features1999
- Recognition by Functional PartsComputer Vision and Image Understanding, 1995
- ASSET-2: real-time motion segmentation and shape trackingIEEE Transactions on Pattern Analysis and Machine Intelligence, 1995
- A Combined Corner and Edge Detector1988
- Determining optical flowArtificial Intelligence, 1981