Semantic Face Compression for Metaverse: A Compact 3D Descriptor Based Approach
- 19 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (9) , 8978-8982
Abstract
The metaverse, a 3D virtual world, requires efficient interactive avatar communication. To achieve this goal, we envision a new metaverse paradigm for virtual avatar faces and develop semantic face compression with compact 3D facial descriptors. The paradigm comprises a compression framework that transmits 3D face descriptors for semantic compression and applications based on the semantic descriptors. The fundamental principle is that the communication of virtual avatar faces primarily emphasizes the conveyance of semantic information. In light of this, the proposed scheme offers the advantages of being highly flexible, efficient, and semantically meaningful. The promise of the proposed paradigm is also demonstrated by performance comparisons with the state-of-the-art video coding standard, Versatile Video Coding. A significant improvement in terms of rate-accuracy performance has been achieved. The proposed scheme is expected to enable numerous applications especially for real-time communication in the metaverse, such as digital human communication based on machine analysis, and to form the cornerstone of interactions.Keywords
All Related Versions
Funding Information
- Shenzhen Science and Technology Program (JCYJ20220530140816037)
- Hong Kong Innovation and Technology Commission [InnoHK Project Centre for Intelligent Multidimensional Data Analysis (CIMDA)]
- Hong Kong Research Grants Council (RGC) of the General Research Fund (11203220, CityU9042957)
- Innovation and Technology Fund (GHP/044/21SZ)
- Alibaba Innovative Research
This publication has 23 references indexed in Scilit:
- AlphaVC: High-Performance and Efficient Learned Video Compression2022
- Evolution of AVS video coding standards: twenty years of innovation and developmentScience China Information Sciences, 2022
- Coarse-To-Fine Deep Video Coding with Hyperprior-Guided Mode Prediction2022
- Towards Ultra Low Bit-Rate Digital Human Character Communication via Compact 3D Face Descriptors2022
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing2021
- M-LVC: Multiple Frames Prediction for Learned Video Compression2020
- DVC: An End-To-End Deep Video Compression Framework2019
- Deep Residual Learning for Image Recognition2016
- FaceNet: A unified embedding for face recognition and clustering2015
- A compression method for clustered bit-vectorsInformation Processing Letters, 1978