MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
Authors:
- Shitao TangSTShitao Tang
- Fuyang ZhangFZFuyang Zhang
- Jiacheng ChenJCJiacheng Chen
- Peng WangPWPeng Wang
- Yasutaka FurukawaYFYasutaka Furukawa
Preprint
Abstract
This paper introduces MVDiffusion, a simple yet effective multi-view image generation method for scenarios where pixel-to-pixel correspondences are available, such as perspective crops from panorama or multi-view images given geometry (depth maps and poses). Unlike prior models that rely on iterative image warping and inpainting, MVDiffusion concurrently generates all images with a global awareness, encompassing high resolution and rich content, effectively addressing the error accumulation prevalent in preceding models. MVDiffusion specifically incorporates a correspondence-aware attention mechanism, enabling effective cross-view interaction. This mechanism underpins three pivotal modules: 1) a generation module that produces low-resolution images while maintaining global correspondence, 2) an interpolation module that densifies spatial coverage between images, and 3) a super-resolution module that upscales into high-resolution outputs. In terms of panoramic imagery, MVDiffusion can generate high-resolution photorealistic images up to 1024$\times$1024 pixels. For geometry-conditioned multi-view image generation, MVDiffusion demonstrates the first method capable of generating a textured map of a scene mesh. The project page is at https://mvdiffusion.github.io.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models2023
- Text2Tex: Text-driven Texture Synthesis via Diffusion Models2023
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- DiffCollage: Parallel Generation of Large Content with Diffusion Models2023
- Pros and cons of GAN evaluation measuresComputer Vision and Image Understanding, 2019
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017