Siwei Ma

Rapid advances in optical imaging have accelerated the emergence of holographic applications. To enable a genuine holographic telepresence experience, the challenge of transmitting massive volumes of holographic data over bandwidth-limited networks must be addressed. This article presents a hologram streaming framework that combines complex-domain rescaling and bitrate allocation tailored to the complex-valued nature of holograms. First, we build a complex-valued rescaling neural network to reduce the required transmission bandwidth. It consists of a downscaling model for reducing hologram volume and a compression-robust upscaling model for recovering high-resolution holograms. The two models are jointly trained to optimize hologram reconstruction quality and are then deployed separately on the server and client. Second, we develop a distortion model that characterizes the different contributions of the real and imaginary (R/I) components to the amplitude distortion of reconstructed holograms. The model is used in a rate-distortion optimization framework to guide optimal R/I bitrate allocation subject to a bandwidth constraint. Experiments on immersive-video datasets show that the proposed framework substantially improves rate-distortion performance; under the BD-rate metric, the baselines require 49.83–222.71% more bitrate to achieve comparable reconstruction quality.

In recent years, with the rapid development of Neural Network-based (NN-based) deep learning algorithms, NN-based video coding tools have been emerging, demonstrating excellent coding performance. However, constrained by limited computational power at the edge or on end-user devices, deploying high-performance NN-based coding tools is a challenging task. This limitation hinders the standardization and widespread application of NN-based coding tools. Recently, NN-based video pre-processing has gained significant attention from both industry and academia. These algorithms are deployed on the cloud and require no modifications to the video codec. Traditional pre-processing methods typically rely on handcrafted designs or natural degradation modeling for quality enhancement. These approaches cannot be jointly optimized with the encoder, making it difficult to perceive the encoding process or adapt to different video content. Moreover, the non-differentiable nature of real codecs makes conventional supervised optimization difficult. Therefore, we propose a one-step reinforcement learning-based framework for video pre-processing model training. We also develop a video pre-processing network capable of supporting 4K@60fps live streaming applications. Experimental results demonstrate that our method achieves significant performance improvements across various video codecs. Compared with codecs such as x265, VVenC, and SVT-AVS3, our method achieves more than a 20% BD-rate (VMAF) improvement. The inference code and pretrained model will be released at: https://github.com/pkuyczhao/RAPNet.

Efficient point cloud compression (PCC) is essential for immersive 3D applications, such as 3D telepresence and remote collaboration, requiring high visual quality despite limited network bandwidth. However, existing PCC methods often prioritize data fidelity while sacrificing perceptual quality, particularly in low-bitrate scenarios. To address these issues, we innovatively propose GS-PCC, the first coding framework that incorporates 3D Gaussian Splatting (3DGS) for PCC. This approach enables high-quality reconstruction with superior perceptual fidelity and optimized bitrate efficiency. Furthermore, GS-PCC provides a unified compression framework that jointly encodes both geometry and attributes. Experimental results demonstrate that GS-PCC significantly outperforms MPEG G-PCC and Google Draco, offering an integrated and efficient solution for human-centric, bandwidth-constrained 3D applications.

With advancement of deep space exploration, vast amounts of image data must be transmitted back to Earth for scientific research. Current deep-space image codecs rely on conventional progressive coding algorithms, but offer limited compression performance. Despite great success on natural images achieved by learning-based compression methods, the high computational complexity restrains their application in the deep space missions and they are unable to cope with packet loss caused by severe noise interference during transmission. Motivated by the urgent need and technical challenges, we take Mars as a representative case and propose a novel image compression and transmission framework that innovatively incorporates learning-based strategies to deliver both low-complexity and error-resilient source coding. To adapt learning-based methods to the stringent constraints and high packet loss of deep space environment, we first establish a new Martian image dataset with high resolution and diversity, and analyze its characteristics to guide the network design. With heterogeneous textures yet synergistic structures as well as higher inter-channel similarity in the feature domain revealed for the Martian images, we develop a Martian Vision Adaptive Transformation Module (MVATM) with efficient low-complexity compression. Furthermore, unlike conventional one-stage training, a novel two-stage training strategy with Joint Channel Training (JCT) is proposed to enhance error resilience. Experimental results and hardware deployment strongly validate that our method achieves a better rate–distortion–complexity (RDC) trade-offs than other advanced learning-based models and significantly outperforms conventional methods in deep space simulation test. Also, the technical strategies proposed herein can offer methodological insights for deep space and other resource-constrained fields.

Subject-customized video generation aims to synthesize videos that adapt to user-specified concepts, ensuring identity consistency while faithfully following textual prompts. The key challenge lies in balancing concept fidelity and prompt following: insufficient adaptation leads to underfitting, losing subject details, whereas excessive adaptation causes overfitting, weakening motion dynamics and scene diversity. To address this, we propose HD-Custom, a test-time fine-tuning framework that leverages hierarchical disentanglement and a coarse-to-fine strategy to efficiently adapt pretrained text-to-video models. Specifically, the coarse stage disentangles category and attribute cues across layers to establish robust subject representations, while the fine stage applies targeted lightweight adaptations to refine appearance details with minimal interference to pretrained priors. A token disentanglement loss and data augmentation are further introduced to enhance attribute–category separation. We also construct HD-Bench, a benchmark explicitly partitioning in-domain and out-of-domain concepts. Extensive experiments on HD-Bench show that HD-Custom achieves a superior balance between concept fidelity and prompt following while requiring substantially fewer trainable parameters and iterations, outperforming state-of-the-art methods.

In recent years, integrating multimodal information, particularly visual and echo data, has shown great promise for improving depth estimation performance. While existing works demonstrate that combining binaural echo features with image attributes can enhance depth estimation, they often use rudimentary feature alignment and fusion methods, failing to fully exploit the complementary nature of cross-modal information and limiting integration effectiveness. To address these challenges, this paper introduces an innovative multimodal fusion framework. First, the framework incorporates a combination of multi-scale self-attention and cross-attention mechanisms, establishing correlations between features and facilitating cohesive interactions between the visual and echo domains. Furthermore, we propose an incremental feature updating mechanism based on Convolutional Gated Recurrent Units (ConvGRU), which implements cascaded iterative optimization, integrating contextual features with the multi-scale fused features from both echo and image modalities. In each iteration, the framework preserves contextual information from previous steps while employing a multi-level loss function to guide result updates. This approach effectively captures spatial structural information and progressively enhances depth estimation accuracy. Comprehensive experimental evaluations on the Replica, Matterport3D and BatVision (BV1) datasets validate the effectiveness of the proposed method. Comparative analyses with state-of-the-art monocular plus echo methods underscore the superior performance achievable through this novel framework.