PU-Dense: Sparse Tensor-Based Point Cloud Geometry Upsampling
- 13 June 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 4133-4148
Abstract
Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and receptive fields. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods.Keywords
Funding Information
- NSF (1747751, 2148382)
This publication has 55 references indexed in Scilit:
- Multi-view Convolutional Neural Networks for 3D Shape Recognition2015
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- 3D ShapeNets: A deep representation for volumetric shapes2015
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Continuous projection for fast L 1 reconstructionACM Transactions on Graphics, 2014
- Immersive 3D TelepresenceComputer, 2014
- Edge-aware point set resamplingACM Transactions on Graphics, 2013
- Consolidation of unorganized point clouds for surface reconstructionACM Transactions on Graphics, 2009
- Parameterization-free projection for geometry reconstructionACM Transactions on Graphics, 2007
- Computing and rendering point set surfacesIEEE Transactions on Visualization and Computer Graphics, 2003