MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3182-3192
Abstract
Neural radiance fields (NeRF) and its subsequent variants have led to remarkable progress in neural rendering. While most of recent neural rendering works focus on objects and small-scale scenes, developing neural rendering methods for city-scale scenes is of great potential in many real-world applications. However, this line of research is impeded by the absence of a comprehensive and high-quality dataset, yet collecting such a dataset over real city-scale scenes is costly, sensitive, and technically infeasible. To this end, we build a large-scale, comprehensive, and high-quality synthetic dataset for city-scale neural rendering researches. Leveraging the Unreal Engine 5 City Sample project, we developed a pipeline to easily collect aerial and street city views, accompanied by ground-truth camera poses and a range of additional data modalities. Flexible controls on environmental factors like light, weather, human and car crowd are also available in our pipeline, supporting the need of various tasks covering city-scale neural rendering and beyond. The resulting pilot dataset, MatrixCity, contains 67k aerial images and 452k street images from two city maps of total size 28km 2 . On top of MatrixCity, a thorough benchmark is also conducted, which not only reveals unique challenges of the task of city-scale neural rendering, but also highlights potential improvements for future works. The dataset and code will be publicly available at the project page: https://city-super.github.io/matrixcity/.Keywords
All Related Versions
Funding Information
- Innovation and Technology Commission
This publication has 34 references indexed in Scilit:
- Capturing, Reconstructing, and Simulating: The UrbanScene3D Dataset2022
- TensoRF: Tensorial Radiance Fields2022
- BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering2022
- HDR-Plenoxels: Self-Calibrating High Dynamic Range Radiance Fields2022
- NeRF for Outdoor Scene Relighting2022
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017
- Large Scale Multi-view Stereopsis Evaluation2014