ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion
Preprint
Abstract
Given a single image of a 3D object, this paper proposes a novel method (named ConsistNet) that is able to generate multiple images of the same object, as if seen they are captured from different viewpoints, while the 3D (multi-view) consistencies among those multiple generated images are effectively exploited. Central to our method is a multi-view consistency block which enables information exchange across multiple single-view diffusion processes based on the underlying multi-view geometry principles. ConsistNet is an extension to the standard latent diffusion model, and consists of two sub-modules: (a) a view aggregation module that unprojects multi-view features into global 3D volumes and infer consistency, and (b) a ray aggregation module that samples and aggregate 3D consistent features back to each view to enforce consistency. Our approach departs from previous methods in multi-view image generation, in that it can be easily dropped-in pre-trained LDMs without requiring explicit pixel correspondences or depth prediction. Experiments show that our method effectively learns 3D consistency over a frozen Zero123 backbone and can generate 16 surrounding views of the object within 40 seconds on a single A100 GPU. Our code will be made available on https://github.com/JiayuYANG/ConsistNetKeywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Explicit Correspondence Matching for Generalizable Neural Radiance FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse ViewsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Delta Denoising Score2023
- Generative Novel View Synthesis with 3D-Aware Diffusion Models2023
- Zero-1-to-3: Zero-shot One Image to 3D Object2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items2022
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004