Physically Guided Generative Adversarial Network for Holographic 3D Content Generation From Multi-View Light Field
- 9 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
- Vol. 14 (2) , 286-298
Abstract
Realizing high-fidelity three-dimensional (3D) scene representation through holography presents a formidable challenge, primarily due to the unknown mechanism of the optimal hologram and huge computational load as well as memory usage. Herein, we propose a Physically Guided Generative Adversarial Network (PGGAN), which is the first generative model to transform the multi-view light field directly to holographic 3D content. PGGAN harmoniously fuses the fidelity of data-driven learning with the rigor of physical optics principles, ensuring a stable reconstruction quality across wide field of view, which is unreachable by current central-view-centric approaches. The proposed framework presents an innovative encoder-generator-discriminator, which is informed by a physical optics model. It benefits from the speed and adaptability of data-driven methods to facilitate rapid learning and effectively transfer to novel scenes, while its physics-based guidance ensures that the generated holograms adhere to holographic standards. A unique, differentiable physical model facilitates end-to-end training, which aligns the generative process with the “holographic space”, thereby improving the quality of the reconstructed light fields. Employing an adaptive loss strategy, PGGAN dynamically adjusts the influence of physical guidance in the initial training stages, later optimizing for reconstruction accuracy. Empirical evaluations reveal PGGAN’s exceptional ability to swiftly generate a detailed hologram in as little as 0.002 seconds, significantly eclipsing current state-of-the-art techniques in speed while maintaining superior angular reconstruction fidelity. These results demonstrate PGGAN’s effectiveness in producing high-quality holograms rapidly from multi-view datasets, advancing real-time holographic rendering significantly.Keywords
Funding Information
- Shenzhen Project (JSGG20220831095602005)
- Natural Science Foundation of Guangdong Province, China (2023A1515012716)
- National Natural Science Foundation of China (62131011, 62205178)
- Major Key Project of PCL (PCL2023A10-3)
This publication has 46 references indexed in Scilit:
- scGPT: toward building a foundation model for single-cell multi-omics using generative AINature Methods, 2024
- Light sheets for continuous-depth holography and three-dimensional volumetric displaysNature Photonics, 2023
- Generating scholarly content with ChatGPT: ethical challenges for medical publishingThe Lancet Digital Health, 2023
- ChatGPT: the future of discharge summaries?The Lancet Digital Health, 2023
- Regarding the quality of disparity estimation from distorted light fieldsSignal Processing: Image Communication, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unitNature Photonics, 2021
- Towards real-time photorealistic 3D holography with deep neural networksNature, 2021
- Ultrathin wide-angle large-area digital 3D holographic display using a non-periodic photon sieveNature Communications, 2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018