Deep Learning-Based Perceptual Video Quality Enhancement for 3D Synthesized View
- 31 January 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (8) , 5080-5094
Abstract
Due to occlusion among views and temporal inconsistency in depth video, spatio-temporal distortion occurs in 3D synthesized video with depth image-based rendering. In this paper, we propose a deep Convolutional Neural Network (CNN)-based synthesized video denoising algorithm to reduce temporal flicker distortion and improve perceptual quality of 3D synthesized video. First, we analyze the spatio-temporal distortion, and model eliminating spatio-temporal distortion as a perceptual video denoising problem. Then, a deep learning-based synthesized video denoising network is proposed, in which a CNN-friendly spatio-temporal loss function is derived from a synthesized video quality metric and integrated with a single image denoising network architecture. Finally, specific schemes, i.e., specific Synthesized Video Denoising Networks (SynVD-Nets), and a general scheme, i.e., General SynVD-Net (GSynVD-Net), based on existing CNN-based denoising models, are developed to handle synthesized video with different distortion levels more effectively. Experimental results show that the proposed SynVD-Net and GSynVD-Net can outperform deep learning-based counterparts and conventional denoising methods, and significantly enhance perceptual quality of 3D synthesized video.Keywords
Funding Information
- National Natural Science Foundation of China (62172400, 61901459)
- Shenzhen Science and Technology Program (JCYJ20180507183823045, JCYJ20200109110410133)
- Guangdong International Science and Technology Cooperative Research Project (2018A050506063)
- Membership of Youth Innovation Promotion Association, Chinese Academy of Sciences (2018392)
- China Postdoctoral Science Foundation (2021T140696)
This publication has 46 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Subjective and Objective Video Quality Assessment of 3D Synthesized Views With Texture/Depth Compression DistortionIEEE Transactions on Image Processing, 2015
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Color video denoising using epitome and sparse codingExpert Systems with Applications, 2015
- Nonlocally Centralized Sparse Representation for Image RestorationIEEE Transactions on Image Processing, 2012
- Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal TransformsIEEE Transactions on Image Processing, 2012
- Information Content Weighting for Perceptual Image Quality AssessmentIEEE Transactions on Image Processing, 2010
- Nonlocal video denoising, simplification and inpainting using discrete regularization on graphsSignal Processing, 2010
- Image Sequence Denoising via Sparse and Redundant RepresentationsIEEE Transactions on Image Processing, 2008
- A Non-Local Algorithm for Image Denoising2005