Multi-View PointNet for 3D Scene Understanding
Authors:
- 1 October 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3995-4003
Abstract
Fusion of 2D images and 3D point clouds is important because information from dense images can enhance sparse point clouds. However, fusion is challenging because 2D and 3D data live in different spaces. In this work, we propose MVPNet (Multi-View PointNet), where we aggregate 2D multi-view image features into 3D point clouds, and then use a point based network to fuse the features in 3D canonical space to predict 3D semantic labels. To this end, we introduce view selection along with a 2D-3D feature aggregation module. Extensive experiments show the benefit of leveraging features from dense images and reveal superior robustness to varying point cloud density compared to 3D-only methods. On the ScanNetV2 benchmark, our MVPNet significantly outperforms prior point cloud based approaches on the task of 3D Semantic Segmentation. It is much faster to train than the large networks of the sparse voxel approach. We provide solid ablation studies to ease the future design of 2D-3D fusion methods and their extension to other tasks, as we showcase for 3D instance segmentation.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters2018
- Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation2018
- 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks2018
- Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models2017
- Mask R-CNN2017
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017
- Pyramid Scene Parsing Network2017
- Deep Residual Learning for Image Recognition2016
- VoxNet: A 3D Convolutional Neural Network for real-time object recognition2015
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012