Perceptual Quality Improvement in Videoconferencing using Keyframes-based GAN
Authors:
Preprint
Abstract
In the latest years, videoconferencing has taken a fundamental role in interpersonal relations, both for personal and business purposes. Lossy video compression algorithms are the enabling technology for videoconferencing, as they reduce the bandwidth required for real-time video streaming. However, lossy video compression decreases the perceived visual quality. Thus, many techniques for reducing compression artifacts and improving video visual quality have been proposed in recent years. In this work, we propose a novel GAN-based method for compression artifacts reduction in videoconferencing. Given that, in this context, the speaker is typically in front of the camera and remains the same for the entire duration of the transmission, we can maintain a set of reference keyframes of the person from the higher-quality I-frames that are transmitted within the video stream and exploit them to guide the visual quality improvement; a novel aspect of this approach is the update policy that maintains and updates a compact and effective set of reference keyframes. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks. This allows the restoration of the high-frequency details lost after the video compression. Experiments show that the proposed approach improves visual quality and generates photo-realistic results even with high compression rates. Code and pre-trained networks are publicly available at https://github.com/LorenzoAgnolucci/Keyframes-GAN.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Multi-Modality Deep Restoration of Extremely Compressed Face VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Generative Adversarial Networks for Image and Video Synthesis: Algorithms and ApplicationsProceedings of the IEEE, 2021
- Quality Prediction on Deep Generative ImagesIEEE Transactions on Image Processing, 2020
- Deep Universal Generative Adversarial Compression Artifact RemovalIEEE Transactions on Multimedia, 2019
- Convolutional Neural Network for Intermediate View Enhancement in Multiview StreamingIEEE Transactions on Multimedia, 2017
- Learning-Based Joint Super-Resolution and Deblocking for a Highly Compressed ImageIEEE Transactions on Multimedia, 2015
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004