Generative Human Video Compression With Multi-Granularity Temporal Trajectory Factorization
- 7 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (1) , 1089-1103
Abstract
In this paper, we propose a novel Multi-granularity Temporal Trajectory Factorization (MTTF) framework for generative human video compression, which holds great potential for bandwidth-constrained human-centric video communication. In particular, the proposed multi-granularity feature factorization strategy can facilitate to implicitly characterize the high-dimensional visual signal into compact motion vectors for representation compactness and further transform these vectors into fine-grained fields for motion expressibility. As such, the coded bit-stream can be entailed with enough visual motion information at the lowest representation cost. Meanwhile, a resolution-expandable generative module is developed with enhanced background stability, such that the proposed framework can be optimized towards higher reconstruction robustness and more flexible resolution adaptation. Experimental results show that proposed method outperforms latest generative models and the state-of-the-art video coding standard Versatile Video Coding (VVC) on both talking-face videos and moving-body videos in terms of both objective and subjective quality. The project page can be found at https://github.com/xyzysz/Extreme-Human-Video-Compression-with-MTTF.Keywords
All Related Versions
Funding Information
- Research Grants Council (RGC) General Research Fund (11200323)
- Innovation and Technology Fund (GHP/044/21SZ)
- City University of Hong Kong Applied Research (9667255)
This publication has 60 references indexed in Scilit:
- Robust Human Matting via Semantic Guidance2023
- Motion Transformer for Unsupervised Image Animation2022
- One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing2021
- Slimmable Compressive Autoencoders for Practical Neural Image Compression2021
- PuppeteerGAN: Arbitrary Portrait Animation With Semantic-Aware Appearance Transformation2020
- DVC: An End-To-End Deep Video Compression Framework2019
- Animating Arbitrary Objects via Deep Motion Transfer2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015