Convolutional Neural Network-Based Synthesized View Quality Enhancement for 3D Video Coding
- 20 July 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 27 (11) , 5365-5377
Abstract
The quality of synthesized view plays an important role in the 3D video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN)-based synthesized view quality enhancement method for 3D high efficiency video coding (HEVC) is proposed. First, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Second, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Third, a new Lagrange multiplier in the rate-distortion cost function is derived to adapt the CNN-based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio and structural similarity index, which significantly outperforms the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (61672443, 61471348, 61571274)
- Hong Kong RGC General Research Fund (9042322 (CityU 11200116), 9042489 (CityU 11206317))
- Hong Kong RGC Early Career Scheme (9048122 (CityU 21211018))
- City University of Hong Kong (7200539/CS)
- Guangdong Natural Science Funds for Distinguished Young Scholar (2016A030306022)
- Project for Shenzhen Science and Technology Development (JSGG20160229202345378)
- Shenzhen International Collaborative Research Project (GJHZ20170314155404913)
- Guangdong Provincial Department of Science and Technology (2017B010110014)
- Guangdong Special Support Program for Youth Science and Technology Innovation Talents (2014TQ01X345)
- Youth Innovation Promotion Association of the Chinese Academy of Sciences (2018392)
- Shandong Natural Science Funds for Distinguished Young Scholar (JQ201614)
This publication has 32 references indexed in Scilit:
- Virtual View Synthesis for Free Viewpoint Video and Multiview Video Compression using Gaussian Mixture ModellingIEEE Transactions on Image Processing, 2017
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image DenoisingIEEE Transactions on Image Processing, 2017
- High-Efficiency 3D Depth Coding Based on Perceptual Quality of Synthesized VideoIEEE Transactions on Image Processing, 2016
- Compression Artifacts Reduction by a Deep Convolutional Network2015
- Depth Image Based View Synthesis: New Insights and Perspectives on Hole Generation and FillingIEEE Transactions on Broadcasting, 2015
- View synthesis distortion elimination filter for depth video coding in 3D video broadcastingMultimedia Tools and Applications, 2014
- View-spatial–temporal post-refinement for view synthesis in 3D video systemsSignal Processing: Image Communication, 2013
- Coding Distortion Elimination of Virtual View Synthesis for 3D Video System: Theoretical Analyses and ImplementationIEEE Transactions on Broadcasting, 2012
- Video Coding Focusing on Block Partitioning and OcclusionIEEE Transactions on Image Processing, 2009
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004