An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement
Authors:
- Dario FuoliDFDario Fuoli
- Zhiwu HuangZHZhiwu Huang
- Danda Pani PaudelDPDanda Pani Paudel
- Luc Van GoolLGLuc Van Gool
- Radu TimofteRadu Timofte
Preprint
Abstract
Video enhancement is a challenging problem, more than that of stills, mainly due to high computational cost, larger data volumes and the difficulty of achieving consistency in the spatio-temporal domain. In practice, these challenges are often coupled with the lack of example pairs, which inhibits the application of supervised learning strategies. To address these challenges, we propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate spatio-temporal information across frames and reduces the need for high complexity networks. Our setting enables learning from unpaired videos in a cyclic adversarial manner, where the proposed recurrent units are employed in all architectures. Efficient training is accomplished by introducing one single discriminator that learns the joint distribution of source and target domain simultaneously. The enhancement results demonstrate clear superiority of the proposed video enhancer over the state-of-the-art methods, in all terms of visual quality, quantitative metrics, and inference speed. Notably, our video enhancer is capable of enhancing over 35 frames per second of FullHD video (1080x1920).Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Unsupervised Low-light Image Enhancement with Decoupled Networks2022
- Towards Unsupervised Deep Image Enhancement With Generative Adversarial NetworkIEEE Transactions on Image Processing, 2020
- Unpaired Image Enhancement Featuring Reinforcement-Learning-Controlled Image Editing SoftwareProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Video Enhancement with Task-Oriented FlowInternational Journal of Computer Vision, 2019
- Recycle-GAN: Unsupervised Video Retargeting2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Multi-Level Video Frame Interpolation: Exploiting the Interaction Among Different LevelsIEEE Transactions on Circuits and Systems for Video Technology, 2013
- Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal TransformsIEEE Transactions on Image Processing, 2012
- Nonlocal video denoising, simplification and inpainting using discrete regularization on graphsSignal Processing, 2010
- Video Denoising Based on a Spatiotemporal Gaussian Scale Mixture ModelIEEE Transactions on Circuits and Systems for Video Technology, 2010