Correspondence Transformers with Asymmetric Feature Learning and Matching Flow Super-Resolution
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 30, 17787-17796
Abstract
This paper solves the problem of learning dense visual correspondences between different object instances of the same category with only sparse annotations. We decompose this pixel-level semantic matching problem into two easier ones: (i) First, local feature descriptors of source and target images need to be mapped into shared semantic spaces to get coarse matching flows. (ii) Second, matching flows in low resolution should be refined to generate accurate point-to-point matching results. We propose asymmetric feature learning and matching flow super-resolution based on vision transformers to solve the above problems. The asymmetric feature learning module exploits a biased cross-attention mechanism to encode token features of source images with their target counterparts. Then matching flow in low resolutions is enhanced by a super-resolution network to get accurate correspondences. Our pipeline is built upon vision transformers and can be trained in an end-to-end manner. Extensive experimental results on several popular benchmarks, such as PF-PASCAL, PF-WILLOW, and SPair-71 K, demonstrate that the proposed method can catch subtle semantic differences in pixels efficiently. Code is available on https://github.com/YXSUNMADMAX/ACTR.Keywords
This publication has 30 references indexed in Scilit:
- Progressive Color Transfer With Dense Semantic CorrespondencesACM Transactions on Graphics, 2019
- Image super-resolution via deterministic-stochastic synthesis and local statistical rectificationACM Transactions on Graphics, 2018
- Pyramid Stereo Matching Network2018
- Proposal Flow: Semantic Correspondences from Object ProposalsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Joint Recovery of Dense Correspondence and Cosegmentation in Two Images2016
- A graph-matching kernel for object categorization2011
- ImageNet: A large-scale hierarchical image database2009
- Medical image registrationPhysics in Medicine & Biology, 2001
- Principal warps: thin-plate splines and the decomposition of deformationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1989
- Speech understanding systemsArtificial Intelligence, 1977