DyLiN: Making Light Field Networks Dynamic
Preprint
Abstract
Light Field Networks, the re-formulations of radiance fields to oriented rays, are magnitudes faster than their coordinate network counterparts, and provide higher fidelity with respect to representing 3D structures from 2D observations. They would be well suited for generic scene representation and manipulation, but suffer from one problem: they are limited to holistic and static scenes. In this paper, we propose the Dynamic Light Field Network (DyLiN) method that can handle non-rigid deformations, including topological changes. We learn a deformation field from input rays to canonical rays, and lift them into a higher dimensional space to handle discontinuities. We further introduce CoDyLiN, which augments DyLiN with controllable attribute inputs. We train both models via knowledge distillation from pretrained dynamic radiance fields. We evaluated DyLiN using both synthetic and real world datasets that include various non-rigid deformations. DyLiN qualitatively outperformed and quantitatively matched state-of-the-art methods in terms of visual fidelity, while being 25 - 71x computationally faster. We also tested CoDyLiN on attribute annotated data and it surpassed its teacher model. Project page: https://dylin2023.github.io .Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Fast Dynamic Radiance Fields with Time-Aware Neural Voxels2022
- R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis2022
- DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle NetworksComputer Graphics Forum, 2021
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New OutlooksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Learning Implicit Fields for Generative Shape Modeling2019
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Single lens stereo with a plenoptic cameraIEEE Transactions on Pattern Analysis and Machine Intelligence, 1992