Prompt2NeRF-PIL: Fast NeRF Generation via Pretrained Implicit Latent
Preprint
Abstract
This paper explores promptable NeRF generation (e.g., text prompt or single image prompt) for direct conditioning and fast generation of NeRF parameters for the underlying 3D scenes, thus undoing complex intermediate steps while providing full 3D generation with conditional control. Unlike previous diffusion-CLIP-based pipelines that involve tedious per-prompt optimizations, Prompt2NeRF-PIL is capable of generating a variety of 3D objects with a single forward pass, leveraging a pre-trained implicit latent space of NeRF parameters. Furthermore, in zero-shot tasks, our experiments demonstrate that the NeRFs produced by our method serve as semantically informative initializations, significantly accelerating the inference process of existing prompt-to-NeRF methods. Specifically, we will show that our approach speeds up the text-to-NeRF model DreamFusion and the 3D reconstruction speed of the image-to-NeRF method Zero-1-to-3 by 3 to 5 times.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- ATT3D: Amortized Text-to-3D Object Synthesis2023
- Zero-1-to-3: Zero-shot One Image to 3D Object2023
- HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion2023
- NeRF-Art: Text-Driven Neural Radiance Fields StylizationIEEE Transactions on Visualization and Computer Graphics, 2023
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Plenoxels: Radiance Fields without Neural Networks2022
- CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation2022
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling2021