SUDS: Scalable Urban Dynamic Scenes
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12375-12385
Abstract
We extend neural radiance fields (NeRFs) to dynamic large-scale urban scenes. Prior work tends to reconstruct single video clips of short durations (up to 10 seconds). Two reasons are that such methods (a) tend to scale linearly with the number of moving objects and input videos because a separate model is built for each and (b) tend to require supervision via 3D bounding boxes and panoptic labels, obtained manually or via category-specific models. As a step towards truly open-world reconstructions of dynamic cities, we introduce two key innovations: (a) we factorize the scene into three separate hash table data structures to efficiently encode static, dynamic, and far-field radiance fields, and (b) we make use of unlabeled target signals consisting of RGB images, sparse LiDAR, off-the-shelf self-supervised 2D descriptors, and most importantly, 2D optical flow. Operationalizing such inputs via photometric, geometric, and feature-metric reconstruction losses enables SUDS to decompose dynamic scenes into the static background, individual objects, and their motions. When combined with our multi-branch table representation, such reconstructions can be scaled to tens of thousands of objects across 1.2 million frames from 1700 videos spanning geospatial footprints of hundreds of kilometers, (to our knowledge) the largest dynamic NeRF built to date. We present qualitative initial results on a variety of tasks enabled by our representations, including novel-view synthesis of dynamic urban scenes, unsupervised 3D instance segmentation, and unsupervised 3D cuboid detection. To compare to prior work, we also evaluate on KITTI and Virtual KITTI 2, surpassing state-of-the-art methods that rely on ground truth 3D bounding box annotations while being 10x quicker to train.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (Extended Abstract)2021
- D-NeRF: Neural Radiance Fields for Dynamic Scenes2021
- NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections2021
- Neural Scene Graphs for Dynamic Scenes2021
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Best-Buddies Similarity for robust template matching2015
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012
- Building Rome in a dayCommunications of the ACM, 2011
- Google Street View: Capturing the World at Street LevelComputer, 2010
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004