GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6796-6807
Abstract
In recent times, the generation of 3D assets from text prompts has shown impressive results. Both 2D and 3D diffusion models can help generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. 2D diffusion models enjoy strong abilities of generalization and fine generation, but 3D consistency is hard to guarantee. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation. A fast 3D object gener-ation framework, named as GaussianDreamer, is proposed, where the 3D diffusion model provides priors for initial-ization and the 2D diffusion model enriches the geometry and appearance. Operations of noisy point growing and color perturbation are introduced to enhance the initialized Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D avatar within 15 minutes on one GPU, much faster than previous methods, while the generated instances can be directly rendered in real time. Demos and code are available at https://taoranyi.com/gaussiandreamer/.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62376102)
This publication has 35 references indexed in Scilit:
- Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation2023
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- UDE: A Unified Driving Engine for Human Motion Generation2023
- MoFusion: A Framework for Denoising-Diffusion-Based Motion Synthesis2023
- Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures2023
- Zero-Shot Text-Guided Object Generation with Dream Fields2022
- CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields2022
- Text2Mesh: Text-Driven Neural Stylization for Meshes2022
- CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation2022
- Diffusion Probabilistic Models for 3D Point Cloud Generation2021