Advancing Learned Video Compression With In-Loop Frame Prediction
- 14 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (5) , 2410-2423
Abstract
Recent years have witnessed an increasing interest in end-to-end learned video compression. Most previous works explore temporal redundancy by detecting and compressing a motion map to warp the reference frame towards the target frame. Yet, it failed to adequately take advantage of the historical priors in the sequential reference frames. In this paper, we propose an Advanced Learned Video Compression (ALVC) approach with the in-loop frame prediction module, which is able to effectively predict the target frame from the previously compressed frames, without consuming any bit-rate . The predicted frame can serve as a better reference than the previously compressed frame, and therefore it benefits the compression performance. The proposed in-loop prediction module is a part of the end-to-end video compression and is jointly optimized in the whole framework. We propose the recurrent and the bi-directional in-loop prediction modules for compressing P-frames and B-frames, respectively. The experiments show the state-of-the-art performance of our ALVC approach in learned video compression. We also outperform the default hierarchical B mode of x265 in terms of PSNR and beat the slowest mode of the SSIM-tuned x265 on MS-SSIM. The project page: https://github.com/RenYang-home/ALVC.Keywords
All Related Versions
Funding Information
- ETH Zurich General Fund
- Alexander von Humboldt-Stiftung
This publication has 50 references indexed in Scilit:
- Optical Flow Estimation Using a Spatial Pyramid Network2017
- Full Resolution Image Compression with Recurrent Neural Networks2017
- Video Traffic Characteristics of Modern Encoding Standards: H.264/AVC with SVC and MVC Extensions and H.265/HEVCThe Scientific World Journal, 2014
- Overview of the High Efficiency Video Coding (HEVC) StandardIEEE Transactions on Circuits and Systems for Video Technology, 2012
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- Motion Refinement Based Progressive Side-Information Estimation for Wyner-Ziv Video CodingIEEE Transactions on Circuits and Systems for Video Technology, 2010
- Overview of the H.264/AVC video coding standardIEEE Transactions on Circuits and Systems for Video Technology, 2003
- Motion vector refinement for high-performance transcodingIEEE Transactions on Multimedia, 1999
- The MPEG video compression algorithmSignal Processing: Image Communication, 1992
- The JPEG still picture compression standardCommunications of the ACM, 1991