MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
Authors:
- Shitao TangSTShitao Tang
- Fuyang ZhangFZFuyang Zhang
- Jiacheng ChenJCJiacheng Chen
- Peng WangPWPeng Wang
- Yasutaka FurukawaYFYasutaka Furukawa
Preprint
Abstract
This paper introduces MVDiffusion, a simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixel correspondences (e.g., perspective crops from a panorama or multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with a global awareness, effectively addressing the prevalent error accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to-image diffusion model, while integrating novel correspondence-aware attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image to a 360-degree view. For multi-view depth-to-image generation, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models2023
- Text2Tex: Text-driven Texture Synthesis via Diffusion Models2023
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- DiffCollage: Parallel Generation of Large Content with Diffusion Models2023
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017