NeRF-Insert: 3D Local Editing with Multimodal Control Signals
Preprint
Abstract
We propose NeRF-Insert, a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control. Unlike previous work that relied on image-to-image models, we cast scene editing as an in-painting problem, which encourages the global structure of the scene to be preserved. Moreover, while most existing methods use only textual prompts to condition edits, our framework accepts a combination of inputs of different modalities as reference. More precisely, a user may provide a combination of textual and visual inputs including images, CAD models, and binary image masks for specifying a 3D region. We use generic image generation models to in-paint the scene from multiple viewpoints, and lift the local edits to a 3D-consistent NeRF edit. Compared to previous methods, our results show better visual quality and also maintain stronger consistency with the original NeRF.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-PromptsACM Transactions on Graphics, 2024
- ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image ContextProceedings of the ACM on Computer Graphics and Interactive Techniques, 2024
- Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields2023
- Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation2023
- Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions2023
- Kandinsky: An Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion2023
- Instant neural graphics primitives with a multiresolution hash encodingACM Transactions on Graphics, 2022
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- The visual hull concept for silhouette-based image understandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 1994