GINA-3D: Learning to Generate Implicit Neural Assets in the Wild
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4913-4926
Abstract
Modeling the 3D world from sensor data for simulation is a scalable way of developing testing and validation environments for robotic learning problems such as autonomous driving. However, manually creating or recreating real-world-like environments is difficult, expensive, and not scalable. Recent generative model techniques have shown promising progress to address such challenges by learning 3D assets using only plentiful 2D images - but still suffer limitations as they leverage either human-curated image datasets or renderings from manually-created synthetic 3D environments. In this paper, we introduce GINA-3D, a generative model that uses real-world driving data from camera and LiDAR sensors to create realistic 3D implicit neural assets of diverse vehicles and pedestrians. Compared to the existing image datasets, the real-world driving setting poses new challenges due to occlusions, lighting-variations and longtail distributions. GINA-3D tackles these challenges by decoupling representation learning and generative modeling into two stages with a learned triplane latent structure, inspired by recent advances in generative modeling of images. To evaluate our approach, we construct a large-scale object-centric dataset containing over 520K images of vehicles and pedestrians from the Waymo Open Dataset, and a new set of 80K images of longtail instances such as construction equipment, garbage trucks, and cable cars. We compare our model with existing approaches and demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.Keywords
All Related Versions
This publication has 74 references indexed in Scilit:
- 1 year, 1000 km: The Oxford RobotCar datasetThe International Journal of Robotics Research, 2016
- A large-scale car dataset for fine-grained categorization and verification2015
- Learning to generate chairs with convolutional neural networks2015
- Unified particle physics for real-time applicationsACM Transactions on Graphics, 2014
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics2010
- Make3D: Learning 3D Scene Structure from a Single Still ImageIEEE Transactions on Pattern Analysis and Machine Intelligence, 2008
- Separating Style and Content with Bilinear ModelsNeural Computation, 2000
- Shape-from-shading: a surveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 1999
- Marching cubes: A high resolution 3D surface construction algorithmACM SIGGRAPH Computer Graphics, 1987