JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds
Preprint
Abstract
In this paper, we propose a novel joint instance and semantic segmentation approach, which is called JSNet, in order to address the instance and semantic segmentation of 3D point clouds simultaneously. Firstly, we build an effective backbone network to extract robust features from the raw point clouds. Secondly, to obtain more discriminative features, a point cloud feature fusion module is proposed to fuse the different layer features of the backbone network. Furthermore, a joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and then the transformed features are further fused with instance features to facilitate instance segmentation. Meanwhile, this module also aggregates instance features into semantic feature space to promote semantic segmentation. Finally, the instance predictions are generated by applying a simple mean-shift clustering on instance embeddings. As a result, we evaluate the proposed JSNet on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset ShapeNet, and compare it with existing approaches. Experimental results demonstrate our approach outperforms the state-of-the-art method in 3D instance segmentation with a significant improvement in 3D semantic prediction and our method is also beneficial for part segmentation. The source code for this work is available at https://github.com/dlinzhao/JSNet.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters2018
- 3D Recurrent Neural Networks with Context Fusion for Point Cloud Semantic Segmentation2018
- SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation2018
- PU-Net: Point Cloud Upsampling Network2018
- SEGCloud: Semantic Segmentation of 3D Point Clouds2017
- Feature Pyramid Networks for Object Detection2017
- Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs2017
- Pyramid Scene Parsing Network2017
- A Field Model for Repairing 3D Shapes2016
- 3D ShapeNets: A deep representation for volumetric shapes2015