Leveraging 2D Data to Learn Textured 3D Mesh Generation
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7495-7504
Abstract
Numerous methods have been proposed for probabilistic generative modelling of 3D objects. However, none of these is able to produce textured objects, which renders them of limited use for practical tasks. In this work, we present the first generative model of textured 3D meshes. Training such a model would traditionally require a large dataset of textured meshes, but unfortunately, existing datasets of meshes lack detailed textures. We instead propose a new training methodology that allows learning from collections of 2D images without any 3D information. To do so, we train our model to explain a distribution of images by modelling each image as a 3D foreground object placed in front of a 2D background. Thus, it learns to generate meshes that when rendered, produce images similar to those in its training set. A well-known problem when generating meshes with deep networks is the emergence of self-intersections, which are problematic for many use-cases. As a second contribution we therefore introduce a new generation process for 3D meshes that guarantees no self-intersections arise, based on the physical intuition that faces should push one another out of the way as they move. We conduct extensive experiments on our approach, reporting quantitative and qualitative results on both synthetic data and natural images. These show our method successfully learns to generate plausible and diverse textured 3D samples for five challenging object classes.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- The shape variational autoencoder: A deep generative model of part‐segmented 3D objectsComputer Graphics Forum, 2017
- GRASSACM Transactions on Graphics, 2017
- Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency2017
- Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks2017
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction2016
- Rethinking the Inception Architecture for Computer Vision2016
- Robust inside-outside segmentation using generalized winding numbersACM Transactions on Graphics, 2013
- Exact and Robust (Self‐)Intersections for Polygonal MeshesComputer Graphics Forum, 2010
- Removing Self Intersections of a Triangular Mesh by Edge Swapping, Edge Hammering, and Face Lifting2009
- Multiscale structural similarity for image quality assessment2004