Textural Detail Preservation Network for Video Frame Interpolation
Open Access
- 13 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 71994-72006
Abstract
The subjective image quality of the Video Frame Interpolation (VFI) result depends on whether image features such as edges, textures and blobs are preserved. With the development of deep learning, various algorithms have been proposed and the objective results of VFI have significantly improved. Moreover, perceptual loss has been used in a method that enhances subjective quality by preserving the features of the image, and as a result, the subjective quality is improved. Despite the quality enhancements achieved in VFI, no analysis has been performed to preserve specific features in the interpolated frames. Therefore, we conducted an analysis to preserve textural detail, such as film grain noise, which can represent the texture of an image, and weak textures, such as droplets or particles. Based on our analysis, we identify the importance of synthesis networks in textural detail preservation and propose an enhanced synthesis network, the Textural Detail Preservation Network (TDPNet). Furthermore, based on our analysis, we propose a Perceptual Training Method (PTM) to address the issue of degraded Peak Signal-to-Noise Ratio (PSNR) when simply applying perceptual loss and to preserve more textural detail. We also propose a Multi-scale Resolution Training Method (MRTM) to address the issue of poor performance when testing datasets with a resolution different from that of the training dataset. The experimental results of the proposed network was outperformed in LPIPS and DISTS on the Vimeo90K, HD, SNU-FILM and UVG datasets compared with the state-of-the-art VFI algorithms, and the subjective results were also outperformed. Furthermore, applying PTM improved PSNR results by an average of 0.293dB compared to simply applying perceptual loss.Keywords
Funding Information
- Institute of Information & Communications Technology Planning & Evaluation (IITP) grant
- Korea governmen (2021-0-00087)
- Development of high-quality conversion technology for SD/HD low-quality media
This publication has 61 references indexed in Scilit:
- Video Frame Interpolation via Adaptive Convolution2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- View Synthesis by Appearance Flow2016
- Deep Residual Learning for Image Recognition2016
- Deep Stereo: Learning to Predict New Views from the World's Imagery2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Enhanced Film Grain Noise Removal and Synthesis for High Fidelity Video CodingIEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2013
- FSIM: A Feature Similarity Index for Image Quality AssessmentIEEE Transactions on Image Processing, 2011
- Multiscale structural similarity for image quality assessment2004
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004