Semantic Reconstruction: Reconstruction of Semantically Segmented 3D Meshes via Volumetric Semantic Fusion
Authors:
- 24 October 2018
- journal article
- conference paper
- Published by Wiley in Computer Graphics Forum
- Vol. 37 (7) , 25-35
Abstract
Semantic segmentation partitions a given image or 3D model of a scene into semantically meaning parts and assigns predetermined labels to the parts. With well‐established datasets, deep networks have been successfully used for semantic segmentation of RGB and RGB‐D images. On the other hand, due to the lack of annotated large‐scale 3D datasets, semantic segmentation for 3D scenes has not yet been much addressed with deep learning. In this paper, we present a novel framework for generating semantically segmented triangular meshes of reconstructed 3D indoor scenes using volumetric semantic fusion in the reconstruction process. Our method integrates the results of CNN‐based 2D semantic segmentation that is applied to the RGB‐D stream used for dense surface reconstruction. To reduce the artifacts from noise and uncertainty of single‐view semantic segmentation, we introduce adaptive integration for the volumetric semantic fusion and CRF‐based semantic label regularization. With these methods, our framework can easily generate a high‐quality triangular mesh of the reconstructed 3D scene with dense (i.e., per‐vertex) semantic labels. Extensive experiments demonstrate that our semantic segmentation results of 3D scenes achieves the state‐of‐the‐art performance compared to the previous voxel‐based and point cloud‐based methods.Keywords
This publication has 25 references indexed in Scilit:
- Learning to Refine Object Segments2016
- PiGraphsACM Transactions on Graphics, 2016
- Learning Deconvolution Network for Semantic Segmentation2015
- Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual informationACM Transactions on Graphics, 2014
- Dense 3D semantic mapping of indoor scenes from RGB-D images2014
- Real-time 3D reconstruction at scale using voxel hashingACM Transactions on Graphics, 2013
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- A volumetric method for building complex models from range images1996
- Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography1987
- Topological structural analysis of digitized binary images by border followingComputer Vision, Graphics, and Image Processing, 1985