Recurrent Flow Update Model Using Image Pyramid Structure for 4K Video Frame Interpolation
Open Access
Abstract
Video frame interpolation (VFI) is a task that generates intermediate frames from two consecutive frames. Previous studies have employed two main approaches to extract the necessary information from both frames: pixel-level synthesis and flow-based methods. However, when synthesizing high-resolution videos using VFI, each approach has its limitations. Pixel-level synthesis based on the transformer architecture requires high complexity to achieve 4K video results. In the case of flow-based methods, forward warping can produce holes where pixels are not allocated, while backward warping approaches struggle to obtain accurate backward flow. Additionally, there are challenges during the training stage; previous works have often generated suboptimal results by training multi-stage model architectures separately. To address these issues, we propose a Recurrent Flow Update (RFU) model trained in an end-to-end manner. We introduce a global flow update module that leverages global information to mitigate the weaknesses of forward flow and gradually correct errors. We demonstrate the effectiveness of our method through several ablation studies. Our approach achieves state-of-the-art performance not only on the XTest and Davis datasets, which have 4K resolution, but also on the SNU-FILM dataset, which features large motions at low resolution.Keywords
Funding Information
- Yonsei Signature Research Cluster Program of 2024 (2024-22-0161, RS-2024-00442120)
- Korea Planning & Evaluation Institute of Industrial Technology (KEIT) (2024-22-0161, RS-2024-00442120)
This publication has 45 references indexed in Scilit:
- Lightweight Single Image Super-Resolution via Efficient Mixture of Transformers and Convolutional NetworksSensors, 2024
- Infrared Image Super-Resolution Network Utilizing the Enhanced Transformer and U-NetSensors, 2024
- An Underwater Image Denoising Method Based on High-Frequency Abrupt Signal Separation and Hybrid Attention MechanismSensors, 2024
- Real-World Video Super-Resolution with a Degradation-Adaptive ModelSensors, 2024
- Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural NetworksSensors, 2024
- Overcoming Periodic Stripe Noise in Infrared Linear Array Images: The Fourier-Assisted Correlative Denoising MethodSensors, 2023
- Joint Video Super-Resolution and Frame Interpolation via Permutation InvarianceSensors, 2023
- Online Video Super-Resolution With Convolutional Kernel Bypass GraftsIEEE Transactions on Multimedia, 2023
- Video Enhancement with Task-Oriented FlowInternational Journal of Computer Vision, 2019
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004