Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion
Open Access
- 13 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Abstract
We present a method for generating Streetscapes—long sequences of views through an on-the-fly synthesized city-scale scene. Our generation is conditioned by language input (e.g., city name, weather), as well as an underlying map/layout hosting the desired trajectory. Compared to recent models for video generation or 3D view synthesis, our method can scale to much longer-range camera trajectories, spanning several city blocks, while maintaining visual quality and consistency. To achieve this goal, we build on recent work on video diffusion, used within an autoregressive framework that can easily scale to long sequences. In particular, we introduce a new temporal imputation method that prevents our autoregressive approach from drifting from the distribution of realistic city imagery. We train our Streetscapes system on a compelling source of data—posed imagery from Google Street View, along with contextual map data—which allows users to generate city views conditioned on any desired city layout, with controllable camera pose.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- Persistent Nature: A Generative Model of Unbounded 3D Worlds2023
- Objaverse: A Universe of Annotated 3D Objects2023
- HOLODIFFUSION: Training a 3D Diffusion Model Using 2D Images2023
- GINA-3D: Learning to Generate Implicit Neural Assets in the Wild2023
- RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation2023
- 3D Neural Field Generation Using Triplane Diffusion2023
- RePaint: Inpainting using Denoising Diffusion Probabilistic Models2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018