Multi-Scale Video Super-Resolution Transformer With Polynomial Approximation
Top Cited Papers
- 19 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (9) , 4496-4506
Abstract
Video super-resolution techniques aim to obtain high-resolution equivalents of existing low-resolution videos through a series of operations. In recent research, transformers have been increasingly popular because of their remarkable abilities in parallel computing and efficient extraction of space-time sequence features from videos. Moreover, combining self-attention and multi-scale methods has yielded excellent results. However, the combination of the two methods has limitations, current up-sampling methods struggle to match the global modeling capacity of self-attention mechanisms. Therefore, this paper proposes three strategies to combine the two methods. Based on the approximation strategy, we first construct a new bilinear up-sampling method for multi-scale acquisition. Convolution and cross-attention techniques are then used to correct and align features at different scales to prevent large deviations in feature extraction at a specific scale, which can affect subsequent feature extraction. Finally, to effectively solve the common computational complexity, C 0 continuity, and neuron death problems of existing activation functions, a new method to construct the activation function is proposed. The cubic spline function is used to construct a new activation function approximating tanh. The new activation function is C 2 continuous, which is piecewise defined by cubic polynomial curves. In this study, better results were achieved on three public video super-resolution test sets: REDS4, Vid4, and Vimeo-90K-T. Experiments demonstrated that the proposed method could provide a new solution for video super-resolution tasks.Keywords
Funding Information
- National Natural Science Foundation of China (62272281, 62007017)
- NSFC Joint Fund with Zhejiang Integration of Informatization and Industrialization through Key Project (U22A2033)
- Youth Innovation Technology Project of Higher School in Shandong Province (2019KJN042)
This publication has 39 references indexed in Scilit:
- Optical Flow Estimation Using a Spatial Pyramid Network2017
- Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation2017
- Deep Video Deblurring for Hand-Held Cameras2017
- Video Super-Resolution via Bidirectional Recurrent Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network2016
- Quantum image scaling using nearest neighbor interpolationQuantum Information Processing, 2014
- On Bayesian Adaptive Video Super ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Bilinear Interpolation2010
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- A method for determining knots in parametric curve interpolationComputer Aided Geometric Design, 1998