Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1581-1891
Abstract
The success of supervised learning requires large- scale ground truth labels which are very expensive, time- consuming, or may need special skills to annotate. To address this issue, many self- or un-supervised methods are developed. Unlike most existing self-supervised methods to learn only 2D image features or only 3D point cloud features, this paper presents a novel and effective self-supervised learning approach to jointly learn both 2D image features and 3D point cloud features by exploiting cross-modality and cross-view correspondences without using any human annotated labels. Specifically, 2D image features of rendered images from different views are extracted by a 2D convolutional neural network, and 3D point cloud features are extracted by a graph convolution neural network. Two types of features are fed into a two-layer fully connected neural network to estimate the cross-modality correspondence. The three networks are jointly trained (i.e. cross-modality) by verifying whether two sampled data of different modalities belong to the same object, meanwhile, the 2D convolutional neural network is additionally optimized through minimizing intra-object distance while maximizing inter-object distance of rendered images in different views (i.e. cross-view). The effectiveness of the learned 2D and 3D features is evaluated by transferring them on five different tasks including multi-view 2D shape recognition, 3D shape recognition, multi-view 2D shape retrieval, 3D shape retrieval, and 3D part-segmentation. Extensive evaluations on all the five different tasks across different datasets demonstrate strong generalization and effectiveness of the learned 2D and 3D features by the proposed self-supervised method.Keywords
All Related Versions
Funding Information
- National Science Foundation
This publication has 30 references indexed in Scilit:
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Learning a Predictable and Generative Vector Representation for Objects2016
- 3D Mesh Labeling via Deep Convolutional Neural NetworksACM Transactions on Graphics, 2015
- VoxNet: A 3D Convolutional Neural Network for real-time object recognition2015
- FaceNet: A unified embedding for face recognition and clustering2015
- 3D deep shape descriptor2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Image Classification with the Fisher Vector: Theory and PracticeInternational Journal of Computer Vision, 2013
- On Visual Similarity Based 3D Model RetrievalComputer Graphics Forum, 2003
- Illumination for computer generated picturesCommunications of the ACM, 1975