Video Frame Interpolation Transformer
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17461-17470
Abstract
Existing methods for video interpolation heavily rely on deep convolution neural networks, and thus suffer from their intrinsic limitations, such as content-agnostic kernel weights and restricted receptive field. To address these issues, we propose a Transformer-based video interpolation framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations. To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video interpolation and extend it to the spatial-temporal domain. Furthermore, we propose a space-time separation strategy to save memory usage, which also improves performance. In addition, we develop a multi-scale frame synthesis scheme to fully realize the potential of Transformers. Extensive experiments demonstrate the proposed model performs favorably against the state-of-the-art methods both quantitatively and qualitatively on a variety of benchmark datasets. The code and models are released at https://github.com/zhshi0816/Video-Frame-Interpolation-Transformer.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation2018
- Non-local Neural Networks2018
- Video Frame Interpolation via Adaptive Separable Convolution2017
- Motion Blur Kernel Estimation via Deep LearningIEEE Transactions on Image Processing, 2017
- JumpACM Transactions on Graphics, 2016
- Deep Residual Learning for Image Recognition2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network2016
- Three-Dimensional Reconstruction of the Digestive Wall in Capsule Endoscopy Videos Using Elastic Video InterpolationIEEE Transactions on Medical Imaging, 2010
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004