FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation
Authors:
- Deepak PathakDPDeepak Pathak
Top Cited Papers
- 1 January 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2070-2081
Abstract
Most modern frame interpolation approaches rely on explicit bidirectional optical flows between adjacent frames, thus are sensitive to the accuracy of underlying flow estimation in handling occlusions while additionally introducing computational bottlenecks unsuitable for efficient deployment. In this work, we propose a flow-free approach that is completely end-to-end trainable for multi-frame video interpolation. Our method, FLAVR, leverages 3D spatio-temporal kernels to directly learn motion properties from unlabeled videos and greatly simplifies the process of training, testing and deploying frame interpolation models. As a result, FLAVR delivers up to 6× speed up compared to the current state-of-the-art methods for multi-frame interpolation while consistently demonstrating superior qualitative and quantitative results compared with prior methods on popular benchmarks including Vimeo-90K, Adobe-240FPS, and GoPro. Finally, we show that frame interpolation is a competitive self-supervised pre-training task for videos via demonstrating various novel applications of FLAVR including action recognition, optical flow estimation, and video object tracking. Code and trained models are provided in the supplementary material.Keywords
All Related Versions
This publication has 72 references indexed in Scilit:
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- Unsupervised Learning of Visual Representations Using Videos2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Object scene flow for autonomous vehicles2015
- Phase-based frame interpolation for video2015
- High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth2014
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- Moving gradientsACM Transactions on Graphics, 2009