An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement
Open Access
- 8 January 2023
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (4) , 1042-1059
Abstract
Video enhancement is a challenging problem, more than that of stills, mainly due to high computational cost, larger data volumes and the difficulty of achieving consistency in the spatio-temporal domain. In practice, these challenges are often coupled with the lack of example pairs, which inhibits the application of supervised learning strategies. To address these challenges, we propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate spatio-temporal information across frames and reduces the need for high complexity networks. Our setting enables learning from unpaired videos in a cyclic adversarial manner, where the proposed recurrent units are employed in all architectures. Efficient training is accomplished by introducing one single discriminator that learns the joint distribution of source and target domain simultaneously. The enhancement results demonstrate clear superiority of the proposed video enhancer over the state-of-the-art methods, in all terms of visual quality, quantitative metrics, and inference speed. Notably, our video enhancer is capable of enhancing over 35 frames per second of FullHD video (1080x1920).Keywords
All Related Versions
Funding Information
- Swiss Federal Institute of Technology Zurich
This publication has 51 references indexed in Scilit:
- A Novel Deep Learning-Based Method of Improving Coding Efficiency from the Decoder-End for HEVC2017
- A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding2016
- Let there be color!ACM Transactions on Graphics, 2016
- Rethinking the Inception Architecture for Computer Vision2016
- Super-resolution: a comprehensive surveyMachine Vision and Applications, 2014
- Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal TransformsIEEE Transactions on Image Processing, 2012
- Optical Flow Guided TV-L1 Video Interpolation and Restoration2011
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- Nonlocal video denoising, simplification and inpainting using discrete regularization on graphsSignal Processing, 2010
- A High-Quality Video Denoising Algorithm Based on Reliable Motion Estimation2010