GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
Preprint
Abstract
Scalable sensor simulation is an important yet challenging open problem for safety-critical domains such as self-driving. Current works in image simulation either fail to be photorealistic or do not model the 3D environment and the dynamic objects within, losing high-level control and physical realism. In this paper, we present GeoSim, a geometry-aware image composition process which synthesizes novel urban driving scenarios by augmenting existing images with dynamic objects extracted from other scenes and rendered at novel poses. Towards this goal, we first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. During simulation, we perform a novel geometry-aware simulation-by-composition procedure which 1) proposes plausible and realistic object placements into a given scene, 2) render novel views of dynamic objects from the asset bank, and 3) composes and blends the rendered image segments. The resulting synthetic images are realistic, traffic-aware, and geometrically consistent, allowing our approach to scale to complex use cases. We demonstrate two such important applications: long-range realistic video simulation across multiple camera sensors, and synthetic data generation for data augmentation on downstream segmentation tasks. Please check https://tmux.top/publication/geosim/ for high-resolution video results.Keywords
All Related Versions
This publication has 46 references indexed in Scilit:
- ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networksComputational Visual Media, 2019
- Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence2018
- Learning Category-Specific Mesh Reconstruction from Image Collections2018
- Layer-Structured 3D Scene Inference via View Synthesis2018
- Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving ScenesInternational Journal of Computer Vision, 2018
- Playing for Data: Ground Truth from Computer Games2016
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- View Synthesis by Appearance Flow2016
- Generative Image Modeling Using Style and Structure Adversarial Networks2016
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes2016