Yifan Zuo

Recent advances in no-reference point cloud quality assessment (NR-PCQA) have been largely driven by deep learning techniques. To further enhance prediction accuracy, existing methods often rely on multi-modal feature fusion. However, the introduction of multimodality ignores the consideration of the model’s generalization ability. Moreover, it often introduces additional computational overhead inference. To address these challenges, we propose a novel NR-PCQA method based on domain-invariant learning framework (DI-PCQA). Specifically, the input point cloud is first projected to obtain a spatial-domain visual representation. To construct different representations with the same origin and semantics, this spatial information is further transformed into a frequency domain representation. Both spatial and frequency features are then fed into a weight-shared encoder to extract feature. Subsequently, an adversarial domain classification task is introduced. We attach the gradient reversal layer to the weight-shared encoder, enabling the encoder to generate features that can confuse the domain classifier. This adversarial training encourages the weight-shared encoder to learn domain-invariant semantic features rather than overfitting to a single data distribution. And the spatial features are simultaneously regressed to quality scores. Finally, DI-PCQA employs a decoupled training-testing strategy. It leverages both spatial and frequency domain inputs to learn domain-invariant representations during training, but performs efficient quality prediction using only spatial information at test time. We conduct extensive tests on public databases and achieve superior performance. Further evaluations confirm its generalization ability and fast inference speed.

Novel view synthesis (NVS) in dynamic scenes faces persistent challenges in memory consumption, model complexity, training efficiency, and rendering quality. Offline methods offer high fidelity but suffer from high memory usage and limited scalability, while online approaches often trade quality for speed and compactness. We propose \textit{Continual Dynamic Neural Graphics Primitives (CD-NGP)}, a continual learning framework that reduces memory overhead and enhances scalability through parameter reuse. To avoid feature interference in dynamic scenes and improve rendering quality, our method combines spatial and temporal hash encodings, which compactly represent scene structures and motion patterns. We also introduce a new dataset comprising multi-view, long-duration ($>1200$ frames) videos with both rigid and non-rigid motion, which is not found in existing benchmarks. CD-NGP is evaluated on public datasets and our long video dataset, demonstrating superior scalability and reconstruction quality. It significantly reduces training memory usage (to $<$14GB) and requires only \textbf{0.4MB/frame} in streaming bandwidth on DyNeRF---substantially lower than most online baselines.%

Point cloud quality assessment (PCQA) aims to establish an objective and unified standard for point cloud evaluation. With the rapid advancement of large language models (LLMs), their application in PCQA has recently gained attention. Existing LLM-based PCQA methods typically adopt static quality description and perceptual information alignment to judge the relevance of each quality level. This overlooks the dynamic nature of human perception, which is progressively refined through multi-level visual stimuli. And the strategy of single-level alignment constrains the variability of the quality description process, thereby resulting in bias. To address this, we propose a dynamic alignment learning framework that simulates progressive quality perception via hierarchical perceptual stimuli. Specifically, to provide a more comprehensive visual information, we capture visual information from two perspectives: from 2D to 3D and from static to dynamic. In addition, a multi-perception learner is utilized to enhance the understanding of these visual information. Subsequently, we employ learnable weights to simulate perceptual stimuli at different levels and embed them into the quality description. During this process, we apply masking to certain perceptual stimuli to illustrate the impact of partial perceptual loss on quality description. Finally, the contrastive learning is introduced to further refine the aligned information under different perceptual stimuli, thereby improving the accuracy of quality assessment. We conduct comprehensive experiments on the public databases, and the results demonstrate that our method achieves state-of-the-art (SOTA) performance. Further analysis confirms the robustness of our approach across various scenarios.

Multi-modal methods based on camera and LiDAR sensors have garnered significant attention in the field of 3D detection. However, many prevalent works focus on single or partial stage fusion, leading to insufficient feature extraction and suboptimal performance. In this paper, we introduce a multi-stage cross-modal fusion 3D detection framework, termed CMF-IOU, to effectively address the challenge of aligning 3D spatial and 2D semantic information. Specifically, we first project the pixel information into 3D space via a depth completion network to get the pseudo points, which unifies the representation of the LiDAR and camera information. Then, a bilateral cross-view enhancement 3D backbone is designed to encode LiDAR points and pseudo points. The first sparse-to-distant (S2D) branch utilizes an encoder-decoder structure to reinforce the representation of sparse LiDAR points. The second residual view consistency (ResVC) branch is proposed to mitigate the influence of inaccurate pseudo points via both the 3D and 2D convolution processes. Subsequently, we introduce an iterative voxel-point aware fine grained pooling module, which captures the spatial information from LiDAR points and textural information from pseudo points in the proposal refinement stage. To achieve more precise refinement during iteration, an intersection over union (IoU) joint prediction branch integrated with a novel proposals generation technique is designed to preserve the bounding boxes with both high IoU and classification scores. Extensive experiments show the superior performance of our method on the KITTI, nuScenes and Waymo datasets. The code is available at https://github.com/pami-zwning/CMF-IOU.

Multi-modal methods based on camera and LiDAR sensors have garnered significant attention in the field of 3D detection. However, many prevalent works focus on single or partial stage fusion, leading to insufficient feature extraction and suboptimal performance. In this paper, we introduce a multi-stage cross-modal fusion 3D detection framework, termed CMF-IOU, to effectively address the challenge of aligning 3D spatial and 2D semantic information. Specifically, we first project the pixel information into 3D space via a depth completion network to get the pseudo points, which unifies the representation of the LiDAR and camera information. Then, a bilateral cross-view enhancement 3D backbone is designed to encode LiDAR points and pseudo points. The first sparse-to-distant (S2D) branch utilizes an encoder-decoder structure to reinforce the representation of sparse LiDAR points. The second residual view consistency (ResVC) branch is proposed to mitigate the influence of inaccurate pseudo points via both the 3D and 2D convolution processes. Subsequently, we introduce an iterative voxel-point aware fine grained pooling module, which captures the spatial information from LiDAR points and textural information from pseudo points in the proposal refinement stage. To achieve more precise refinement during iteration, an intersection over union (IoU) joint prediction branch integrated with a novel proposals generation technique is designed to preserve the bounding boxes with both high IoU and classification scores. Extensive experiments show the superior performance of our method on the KITTI, nuScenes and Waymo datasets.

Infrared and visible image fusion aims to generate fused images with rich textures and clear target representations. Existing methods generally assume high-quality input images, thus overlooking issues such as reduced contrast and loss of details in visible images under low-light conditions. The naive enhance-then-fuse strategy cannot perform fuse-oriented image enhancement, which always reaches a sub-optimal result. To address this challenge, we propose a perceptual transform fusion of infrared and visible images, which simultaneously optimizes low-light enhancement and image fusion. Specifically, to improve computational efficiency and optimize key feature representations while suppressing noise interactions caused by lighting variations, we introduce a lightweight adaptive sparse Transformer block (ASTBlock). This model adaptively integrates sparse and dense attention mechanisms to enhance feature representations and employs a feed-forward network to eliminate redundant information, thereby ensuring the quality of image fusion. Subsequently, to retain significant details while reducing the impact of noise introduced by low-light enhancement, we incorporate discrete wavelet transform (DWT) for feature decomposition and fusion, further enhancing the representation capability and feature preservation of fused images. Meanwhile, to tackle the issues of insufficient contrast and hidden details in low-light conditions, we design an illumination perception module and an illumination consistency loss to improve the contrast and clarity of fused images. Experimental results on multiple public benchmark datasets for quality assessment and downstream tasks, e.g., pedestrian detection, demonstrate that our method significantly outperforms the state-of-the-art (SOTA) methods. The code is available at https://github.com/hinmouc/PIVFusion.

Recently, the single image super-resolution based on implicit image function is a hot topic, which learns a universal model for arbitrary upsampling scales. By contrast, color-guided depth map super-resolution is less explored based on implicit function learning. The related research faces three questions. First, is it also necessary and applicable to fuse the depth feature and the color feature in the encoder with continuous upsampling scales? Second, is the scale information in the encoder as important as that in the decoder? Third, how to efficiently and effectively model the affinity of location distance and content similarity within cross domains in the decoder? This paper proposes a transformer-based network to answer the above questions, which includes a depth super-resolution branch and a guidance extraction branch. Specifically, in the encoder, the effective implicit cross transformer is designed to fuse the guidance from the color feature with continuous coordinate mapping. In addition, the unrelated guidance is filtered out by correlation evaluation in the high-dimension feature space. Unlike the scale only introduced in the decoder, this paper additionally embeds the scale into the position encoding and the feed-forward network in the encoder to learn the scale-aware feature representation. In the decoder, the high-resolution depth feature is reconstructed by using the internal prior and the external guidance. The internal prior is implemented by implicit self-attention in the depth super-resolution branch, and the external guidance is exploited via implicit cross-attention between both branches. Finally, the above decoded features are complementary to generate the high-resolution depth map. The sufficient experiments on the synthetic and real datasets for in-distribution and out-of-distribution upsampling scales validate the improved performance. The code and the models are public via https://github.com/NaNRan13/GIDF.