Gaussian Splatting with NeRF-based Color and Opacity
Authors:
- Dawid MalarzDMDawid Malarz
- Weronika SmolakWSWeronika Smolak
- Jacek TaborJTJacek Tabor
- Sławomir TadejaSTSławomir Tadeja
- Przemysław SpurekPSPrzemysław Spurek
Preprint
Abstract
Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar render quality with faster training and inference as it does not need neural networks to work. We encode information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model Viewing Direction Gaussian Splatting (VDGS) that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and neural network, which takes parameters of Gaussian and viewing direction to produce changes in color and opacity. Consequently, our model better describes shadows, light reflections, and transparency of 3D objects.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- MultiPlaneNeRF: Neural radiance field with non-trainable representationExpert Systems with Applications, 2025
- GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces2024
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering2024
- ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting2023
- Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction2023
- Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction2022
- Plenoxels: Radiance Fields without Neural Networks2022
- Continuous conditional random field convolution for point cloud segmentationPattern Recognition, 2022
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction2016