Ultra-Low Bitrate Face Video Compression Based on Conversions From 3D Keypoints to 2D Motion Map
- 20 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6850-6864
Abstract
How to compress face video is a crucial problem for a series of online applications, such as video chat/conference, live broadcasting and remote education. Compared to other natural videos, these face-centric videos owning abundant structural information can be compactly represented and high-quality reconstructed via deep generative models, such that the promising compression performance can be achieved. However, the existing generative face video compression schemes are faced with the inconsistency between the 3D facial motion in the physical world and the face content evolution in the 2D view. To solve this drawback, we propose a 3D-Keypoint-and-2D-Motion based generative method for Face Video Compression, namely FVC-3K2M, which can well ensure perceptual compensation and visual consistency between motion description and face reconstruction. In particular, the temporal evolution of face video can be characterized into separate 3D keypoints from the global and local perspectives, entailing great coding flexibility and accurate motion representation. Moreover, a cascade motion conversion mechanism is further proposed to internally convert 3D keypoints to 2D dense motion, enforcing the face video reconstruction to be perceptually realistic. Finally, an adaptive reference frame selection scheme is developed to enhance the adaptation of various temporal movements. Experimental results show that the proposed scheme can realize reliable video communication in the extremely limited bandwidth, e.g., 2 kbps. Compared to the state-of-the-art video coding standards and the latest face video compression methods, extensive comparisons demonstrate that our proposed scheme achieves superior compression performance in terms of multiple quality evaluations.Keywords
Funding Information
- “Leading Goose” Research and Development Program of Zhejiang Province (2024C01101)
- National Natural Science Foundation of China (62025101, 62088102)
- Fundamental Research Funds for the Central Universities
This publication has 51 references indexed in Scilit:
- SliderGAN: Synthesizing Expressive Face Images by Sliding 3D Blendshape ParametersInternational Journal of Computer Vision, 2020
- GANimation: Anatomically-Aware Facial Animation from a Single Image2018
- X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose Codes2018
- A 3D Morphable Model Learnt from 10,000 Faces2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- A 3D Face Model for Pose and Illumination Invariant Face Recognition2009
- Two-Frame Motion Estimation Based on Polynomial Expansion2003
- Head pose computation for very low bit-rate video coding1995
- Model-based image coding advanced video coding techniques for very low bit-rate applicationsProceedings of the IEEE, 1995
- A switched model-based coder for video signalsIEEE Transactions on Circuits and Systems for Video Technology, 1994