IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration
- 14 October 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (4) , 12323-12330
Abstract
The existing state-of-the-art point descriptor relies on structure information only, which omits the texture information. However, texture information is crucial for our humans to distinguish a scene part. Moreover, the current learning-based point descriptors are all black boxes which are unclear how the original points contribute to the final descriptor. This paper proposes a new multimodal fusion method to generate a point cloud registration descriptor by considering structure and texture information. Specifically, a novel attention-fusion module is designed to extract the weighted texture information for descriptor extraction. In addition, we propose an interpretable module to explain our neural network by visually showing the original points contributing to the final descriptor. We use the descriptor's channel value as the loss to backpropagate to the target layer and consider the gradient as the significance of this point to the final descriptor. This paper moves one step further to explainable deep learning in the registration task. Comprehensive experiments on 3DMatch, 3DLoMatch and KITTI demonstrate that the multimodal fusion descriptor achieves state-of-the-art accuracy and improves the descriptor's distinctiveness. We also demonstrate our interpretable module in explaining the registration descriptor extraction.Keywords
All Related Versions
Funding Information
- Shanghai AI Laboratory
- National Natural Science Foundation of China (62271237, 61901197)
- Natural Science Foundation of Jiangxi Province (20212BAB202011)
This publication has 22 references indexed in Scilit:
- PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors2018
- Learning localized features in 3D CAD models for manufacturability analysis of drilled holesComputer Aided Geometric Design, 2018
- Colored Point Cloud Registration Revisited2017
- Learning Compact Geometric Features2017
- 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions2017
- A Coarse-to-Fine Algorithm for Registration in 3D Street-View Cross-Source Point Clouds2016
- Color-aware surface registrationComputers & Graphics, 2016
- Real Time Complete Dense Depth Reconstruction for a Monocular Camera2016
- Ensemble of shape functions for 3D object classification2011
- Fast Point Feature Histograms (FPFH) for 3D registration2009