Deep Spatial-Angular Regularization for Light Field Imaging, Denoising, and Super-Resolution
- 8 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (10) , 6094-6110
Abstract
Coded aperture is a promising approach for capturing the 4-D light field (LF), in which the 4-D data are compressively modulated into 2-D coded measurements that are further decoded by reconstruction algorithms. The bottleneck lies in the reconstruction algorithms, resulting in rather limited reconstruction quality. To tackle this challenge, we propose a novel learning-based framework for the reconstruction of high-quality LFs from acquisitions via learned coded apertures. The proposed method incorporates the measurement observation into the deep learning framework elegantly to avoid relying entirely on data-driven priors for LF reconstruction. Specifically, we first formulate the compressive LF reconstruction as an inverse problem with an implicit regularization term. Then, we construct the regularization term with a deep efficient spatial-angular separable convolutional sub-network in the form of local and global residual learning to comprehensively explore the signal distribution free from the limited representation ability and inefficiency of deterministic mathematical modeling. Furthermore, we extend this pipeline to LF denoising and spatial super-resolution, which could be considered as variants of coded aperture imaging equipped with different degradation matrices. Extensive experimental results demonstrate that the proposed methods outperform state-of-the-art approaches to a significant extent both quantitatively and qualitatively, i.e., the reconstructed LFs not only achieve much higher PSNR/SSIM but also preserve the LF parallax structure better on both real and synthetic LF benchmarks. The code will be publicly available at https://github.com/MantangGuo/DRLF .Keywords
Funding Information
- Hong Kong Research Grants Council (9048123 (CityU 21211518), 9042820 (CityU 11219019))
- Basic Research General Program of Shenzhen Municipality (JCYJ20190808183003968)
This publication has 59 references indexed in Scilit:
- Light Field Reconstruction Using Sparsity in the Continuous Fourier DomainACM Transactions on Graphics, 2014
- Variational Light Field Analysis for Disparity Estimation and Super-ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Compressive light field photography using overcomplete dictionaries and optimized projectionsACM Transactions on Graphics, 2013
- Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter2013
- Spatial and Angular Variational Super-Resolution of 4D Light Fields2012
- Programmable Aperture Camera Using LCoS2010
- Programmable aperture photographyACM Transactions on Graphics, 2008
- High performance imaging using large camera arraysACM Transactions on Graphics, 2005
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Light field rendering1996