Weimin Lei

The application of animation models in facial video compression has yielded significant coding gains, particularly at ultra-low bitrates. Despite notable advancements, research on portrait video scenes, especially in half-body and full-body contexts, remains underexplored. The mapping motion features and dynamic regions is often imprecise, leading to inaccurate motion parameters. Additionally, the reconstruction of occluded background is frequently suboptimal, as occluded regions lack ground-truth data. To this end, we propose a learning-based portrait video compression (LPVC) framework for half-body and full-body portrait videos with static backgrounds. Specifically, we first design a keypoint detector driven by human parsing to integrate semantic properties into the animation model. This facilitates richer features and enhances the motion features for dynamic areas. We further devise a background incremental coding scheme to reconstruct high-quality occluded backgrounds. The scheme incorporates background occlusion calculation and compensation modules to process the newly revealed background pixels between successive frames, eliminating redundant background transmission. Finally, a spatio-temporal portrait background fusion generative adversarial network (SPBF-GAN) is devised to learn spatio-temporal differential representations of videos. Experimental results demonstrate that our proposed scheme achieves satisfying perceptual performance at ultra-low bitrates and exhibits higher semantic fidelity in semantic analysis tasks such as human parsing. Code is available at: https://github.com/Chen8023/LPVC .

To address the dual challenges of inadequate deep semantic feature representation and limited data diversity in bi-temporal remote sensing change detection (RSCD), we propose a collaborative optimization framework (SRA-VFM) integrating customized data augmentation and hierarchical feature interpretation. SRA-VFM comprises three core components: slice-reassemble augmentation (SRA), a dual-stream feature encoding-decoding network, and a multi-task segmentation head. The SRA module synthesizes diverse training samples through random slicing and semantic reassembly while preserving local feature consistency. The dual-stream encoder, based on the FastSAM pre-trained model, incorporates a top-down feature adapter to align pre-trained features with remote sensing data distributions via cross-level fusion and low-dimensional semantic mapping. The bottom-up decoder leverages semantically aligned pyramid features for progressive upsampling, restoring high-resolution spatial details and enhancing multi-scale representation. The multi-task head jointly optimizes change region detection and edge refinement, accelerating convergence and improving boundary localization. Extensive experiments on 5 benchmark datasets (LEVIR-CD, WHU-CD, CLCD, S2Looking, SYSU-CD) demonstrate SRA-VFM’s superiority over state-of-the-art (SOTA) methods, achieving mF1/mIoU of 96.01%/92.55%, 97.13%/94.53%, 88.91%/81.21%, 83.13%/48.91%, and 88.36%/79.67% respectively. Code will be publicly available upon publication.

End-to-end matting methods such as MODNet, Robust Video Matting (RVM), and VideoMatt have significantly improved the efficiency of matting. However, there are still deficiencies in handling foreground details and similar foreground-background scenarios, leaving considerable room for improvement in accuracy. We propose a real-time video portrait matting method based on large-kernel convolutions, achieving high-precision real-time video portrait matting through the use of large-kernel convolutions, wavelet transforms, and reparameterization techniques. The network employs large-kernel convolutions to enhance the encoding and decoding capabilities of image features. In addition, the wavelet transform is used to pass image information to the decoding network, providing more useful information. Finally, reparameterization technology is employed to fuse multiple convolution branches with large kernels in parallel, effectively reducing computational burden and speeding up inference. The model can process high-definition video at 94 frames per second (FPS) and 4K video at 86 FPS on an Nvidia RTX 3090 graphics processing unit. Performance metrics on multiple public video matting datasets demonstrate that the model has reached an advanced level.

Deep neural video compression codecs have shown great promise in recent years. However, there are still considerable challenges for ultra-low bitrate video coding. Inspired by recent diffusion models for image and video compression attempts, we attempt to leverage diffusion models for ultra-low bitrate portrait video compression. In this paper, we propose a predictive portrait video compression method that leverages the temporal prediction capabilities of diffusion models. Specifically, we develop a temporal diffusion predictor based on a conditional latent diffusion model, with the predicted results serving as decoded frames. We symmetrically integrate a temporal diffusion predictor at the encoding and decoding side, respectively. When the perceptual quality of the predicted results in encoding end falls below a predefined threshold, a new frame sequence is employed for prediction. While the predictor at the decoding side directly generates predicted frames as reconstruction based on the evaluation results. This symmetry ensures that the prediction frames generated at the decoding end are consistent with those at the encoding end. We also design an adaptive coding strategy that incorporates frame quality assessment and adaptive keyframe control. To ensure consistent quality of subsequent predicted frames and achieve high perceptual reconstruction, this strategy dynamically evaluates the visual quality of the predicted results during encoding, retains the predicted frames that meet the quality threshold, and adaptively adjusts the length of the keyframe sequence based on motion complexity. The experimental results demonstrate that, compared with the traditional video codecs and other popular methods, the proposed scheme provides superior compression performance at ultra-low bitrates while maintaining competitiveness in visual effects, achieving more than 24% bitrate savings compared with VVC in terms of perceptual distortion.