UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene
Top Cited Papers
- 19 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (4) , 2045-2057
Abstract
Photorealistic stylization of 3D scenes aims to generate photorealistic images from arbitrary novel views according to a given style image, while ensuring consistency when rendering video from different viewpoints. Some existing stylization methods using neural radiance fields can effectively predict stylized scenes by combining the features of the style image with multi-view images to train 3D scenes. However, these methods generate novel view images that contain undesirable artifacts. In addition, they cannot achieve universal photorealistic stylization for a 3D scene. Therefore, a stylization image needs to retrain a 3D scene representation network based on a neural radiation field. We propose a novel photorealistic 3D scene stylization transfer framework to address these issues. It can realize photorealistic 3D scene style transfer with a 2D style image for novel view video rendering. We first pre-trained a 2D photorealistic style transfer network, which can satisfy the photorealistic style transfer between any content image and style image. Then, we use voxel features to optimize a 3D scene and obtain the geometric representation of the scene. Finally, we jointly optimize a hypernetwork to realize the photorealistic style transfer of arbitrary style images. In the transfer stage, we use a pre-trained 2D photorealistic network to constrain the photorealistic style of different views and different style images in the 3D scene. The experimental results show that our method not only realizes the 3D photorealistic style transfer of arbitrary style images, but also outperforms the existing methods in terms of visual quality and consistency. Project page: https://semchan.github.io/UPST_NeRF/ .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62172061)
- National Key R&D Program of China (2020YFB1711800, 2020YFB1707900)
This publication has 63 references indexed in Scilit:
- Unified Implicit Neural Stylization2022
- ARF: Artistic Radiance Fields2022
- CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer2022
- Neural Point-Based Graphics2020
- Joint Bilateral Learning for Real-Time Universal Photorealistic Style Transfer2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Semantic View Synthesis2020
- A Closed-Form Solution to Photorealistic Image Stylization2018
- Learning Category-Specific Mesh Reconstruction from Image Collections2018
- Microsoft COCO: Common Objects in Context2014