Zhanjie Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Zhanjie Zhang has a total of 69 co-authors. Here are the top co-authors by number of shared publications:
- Lei Zhao
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
10 shared publications Quanwei Zhang
Zhejiang University
6 shared publications- Wei Xing
College of Computer Science and Technology, Zhejiang University, Hangzhou, China
6 shared publications - Huaizhong Lin
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
4 shared publications - Jiakai Sun
Zhejiang University
4 shared publications Lei Zhao
4 shared publications
Latest Publications
- 3 October 2026
- Published by Springer Nature
- p. 111-127
InnoText: A Unified Model for Visual Text Generation and Editing
- 3 October 2026
- Published by Springer Nature
- p. 111-127
InnoText: A Unified Model for Visual Text Generation and Editing
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 5858-5870
Fast and Robust Deformable 3D Gaussian Splatting
3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 5858-5870
Fast and Robust Deformable 3D Gaussian Splatting
3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.
- 27 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 4494-4508
SMFormer: Empowering Self-Supervised Stereo Matching via Foundation Models and Data Augmentation
Recent self-supervised stereo matching methods have made significant progress. They typically rely on the photometric consistency assumption, which presumes corresponding points across views share the same appearance. However, this assumption could be compromised by real-world disturbances, resulting in invalid supervisory signals and a significant accuracy gap compared to supervised methods. To address this issue, we propose SMFormer, a framework integrating more reliable self-supervision guided by the Vision Foundation Model (VFM) and data augmentation. We first incorporate the VFM with the Feature Pyramid Network (FPN), providing a discriminative and robust feature representation against disturbance in various scenarios. We then devise an effective data augmentation mechanism that ensures robustness to various transformations. The data augmentation mechanism explicitly enforces consistency between learned features and those influenced by illumination variations. Additionally, it regularizes the output consistency between disparity predictions of strong augmented samples and those generated from standard samples. Experiments on multiple mainstream benchmarks demonstrate that our SMFormer achieves state-of-the-art (SOTA) performance among self-supervised methods and even competes on par with supervised ones. Remarkably, in the challenging Booster benchmark, SMFormer even outperforms some SOTA supervised methods, such as CFNet.
- 27 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 4494-4508
SMFormer: Empowering Self-Supervised Stereo Matching via Foundation Models and Data Augmentation
Recent self-supervised stereo matching methods have made significant progress. They typically rely on the photometric consistency assumption, which presumes corresponding points across views share the same appearance. However, this assumption could be compromised by real-world disturbances, resulting in invalid supervisory signals and a significant accuracy gap compared to supervised methods. To address this issue, we propose SMFormer, a framework integrating more reliable self-supervision guided by the Vision Foundation Model (VFM) and data augmentation. We first incorporate the VFM with the Feature Pyramid Network (FPN), providing a discriminative and robust feature representation against disturbance in various scenarios. We then devise an effective data augmentation mechanism that ensures robustness to various transformations. The data augmentation mechanism explicitly enforces consistency between learned features and those influenced by illumination variations. Additionally, it regularizes the output consistency between disparity predictions of strong augmented samples and those generated from standard samples. Experiments on multiple mainstream benchmarks demonstrate that our SMFormer achieves state-of-the-art (SOTA) performance among self-supervised methods and even competes on par with supervised ones. Remarkably, in the challenging Booster benchmark, SMFormer even outperforms some SOTA supervised methods, such as CFNet.
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 5346-5355
RAFed: Responsive Augmentation and Approximate Update Method for Federated Learning with Non-IID Data
Federated learning is a distributed collaborative training framework that enables multiple clients to share model updates and jointly train deep neural networks without exchanging raw data. Although extensive research has explored data augmentation techniques in federated settings, the naturally non-IID data distributions among clients render blind augmentation prone to severe degradation of the learned model. To solve this problem, we suggest Responsive Augmentation and Approximate Update Method for Fed erated Learning with Non-IID Data (RAFed), aimed at alleviating feature shift in client samples. We leverage a Responsive Augmentation Method to accumulate shared data augmentation policy knowledge through local learning, guiding the policy gradient to consider the impact of data augmentation on unseen local data, and employ an Approximate Update Mechanism to reduce communication costs and achieve efficient policy search. To improve the adaptability of data augmentation policies to local data distributions, we introduce a Dynamic Adaptive Method for searching personalized augmentation policies tailored to heterogeneous clients. Experiments on four popular datasets show that RAFed achieves superior test accuracy and lower communication costs compared to related baselines while providing privacy advantages. The code is available via https://github.com/anonymously123-stcak/RAFed.
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 5346-5355
Open AccessRAFed: Responsive Augmentation and Approximate Update Method for Federated Learning with Non-IID Data
Federated learning is a distributed collaborative training framework that enables multiple clients to share model updates and jointly train deep neural networks without exchanging raw data. Although extensive research has explored data augmentation techniques in federated settings, the naturally non-IID data distributions among clients render blind augmentation prone to severe degradation of the learned model. To solve this problem, we suggest Responsive Augmentation and Approximate Update Method for Fed erated Learning with Non-IID Data (RAFed), aimed at alleviating feature shift in client samples. We leverage a Responsive Augmentation Method to accumulate shared data augmentation policy knowledge through local learning, guiding the policy gradient to consider the impact of data augmentation on unseen local data, and employ an Approximate Update Mechanism to reduce communication costs and achieve efficient policy search. To improve the adaptability of data augmentation policies to local data distributions, we introduce a Dynamic Adaptive Method for searching personalized augmentation policies tailored to heterogeneous clients. Experiments on four popular datasets show that RAFed achieves superior test accuracy and lower communication costs compared to related baselines while providing privacy advantages. The code is available via https://github.com/anonymously123-stcak/RAFed.
- Open Access
- 19 March 2026
- Published by MDPI AG in Applied Sciences
- Vol. 16 (6) , 2969
CSCGAN: Cross-Space Contrastive Learning for Blind Image Inpainting
Existing general image inpainting works require the user to customize a mask to indicate the region to be inpainted. However, the mask is often hard to calibrate accurately in real-world applications, e.g., graffiti removal. Blind image inpainting aims to automatically restore the degraded image into the visually reasonable one without a priori mask to indicate the area to be repaired. So far, most proposed blind inpainting methods convert the task into general inpainting by predicting the mask before inpainting. However, these methods are highly dependent on mask prediction results, which may produce inferior inpainting results if the prediction is inaccurate. To address this issue, we propose a two-stage blind inpainting framework with two novel designs: (1) cross-space contrastive learning, to remove the noise in the degraded images and realize the automatic inpainting in the latent space by reducing the distance of the degraded images and the corresponding complete images in the latent space; and (2) mask-aware adversarial training, to minimize the mutual information between the inpainted feature and the noise. Extensive experiments prove that our blind inpainting framework performs better on multiple datasets than the state-of-the-art methods.
- 19 March 2026
- Published by MDPI AG in Applied Sciences
- Vol. 16 (6) , 2969
Open AccessCSCGAN: Cross-Space Contrastive Learning for Blind Image Inpainting
Existing general image inpainting works require the user to customize a mask to indicate the region to be inpainted. However, the mask is often hard to calibrate accurately in real-world applications, e.g., graffiti removal. Blind image inpainting aims to automatically restore the degraded image into the visually reasonable one without a priori mask to indicate the area to be repaired. So far, most proposed blind inpainting methods convert the task into general inpainting by predicting the mask before inpainting. However, these methods are highly dependent on mask prediction results, which may produce inferior inpainting results if the prediction is inaccurate. To address this issue, we propose a two-stage blind inpainting framework with two novel designs: (1) cross-space contrastive learning, to remove the noise in the degraded images and realize the automatic inpainting in the latent space by reducing the distance of the degraded images and the corresponding complete images in the latent space; and (2) mask-aware adversarial training, to minimize the mutual information between the inpainted feature and the noise. Extensive experiments prove that our blind inpainting framework performs better on multiple datasets than the state-of-the-art methods.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 12195-12206
SAAST: Artistic Style Transfer via Pre-Trained Stable Diffusion With Step-Aware Content and Style Learning
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 12195-12206
SAAST: Artistic Style Transfer via Pre-Trained Stable Diffusion With Step-Aware Content and Style Learning
- 27 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (4) , 5274-5287
IP-Controller: Decomposition and Optimization of Cross-Attention Maps for Accurate Subject-Driven Text-to-Image Diffusion Generation
Although large pretrained stable diffusion (SD) models can generate high-quality images from prompts, they cannot generate images that are consistent with the fine-grained characteristics of a specific identity V ∗ (e.g., an anime character). Subject-driven generation focuses on exploring and leveraging the prior knowledge within a model to achieve the goals of ID and context preservation. There have been efforts, such as DreamBooth, to conduct subject-driven generation; however, they suffer from ID and context mistakes. An ID mistake means a feature loss of V ∗, and a context mistake means that the generated image does not align with the given prompt. To rectify these problems, in this paper, we propose masked fine-tuning for efficient feature learning of V ∗, then propose IP-Controller for decomposing and optimizing cross-attention maps of V ∗ and prompt words other than V ∗. Specifically, we generate the cross-attention map using a vanilla input prompt and decompose it into an ID cross-attention map (matching V ∗) and a context cross-attention map (matching prompt words other than V ∗). Next, we generate fitter ID and context cross-attention maps on the basis of the input ID and context prompts, respectively. We optimize the ID and context cross-attention maps with the fitter ID and context cross-attention maps, respectively, so that the diffusion process pays fitter attention for specific contents. Experiments show that IP-Controller correctly integrates the core features of V ∗ and the semantic context of the prompt words other than V ∗ and generates high-quality images for the given prompt.
- 27 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (4) , 5274-5287
IP-Controller: Decomposition and Optimization of Cross-Attention Maps for Accurate Subject-Driven Text-to-Image Diffusion Generation
Although large pretrained stable diffusion (SD) models can generate high-quality images from prompts, they cannot generate images that are consistent with the fine-grained characteristics of a specific identity V ∗ (e.g., an anime character). Subject-driven generation focuses on exploring and leveraging the prior knowledge within a model to achieve the goals of ID and context preservation. There have been efforts, such as DreamBooth, to conduct subject-driven generation; however, they suffer from ID and context mistakes. An ID mistake means a feature loss of V ∗, and a context mistake means that the generated image does not align with the given prompt. To rectify these problems, in this paper, we propose masked fine-tuning for efficient feature learning of V ∗, then propose IP-Controller for decomposing and optimizing cross-attention maps of V ∗ and prompt words other than V ∗. Specifically, we generate the cross-attention map using a vanilla input prompt and decompose it into an ID cross-attention map (matching V ∗) and a context cross-attention map (matching prompt words other than V ∗). Next, we generate fitter ID and context cross-attention maps on the basis of the input ID and context prompts, respectively. We optimize the ID and context cross-attention maps with the fitter ID and context cross-attention maps, respectively, so that the diffusion process pays fitter attention for specific contents. Experiments show that IP-Controller correctly integrates the core features of V ∗ and the semantic context of the prompt words other than V ∗ and generates high-quality images for the given prompt.
- 6 September 2025
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 161
Personalized text-to-image generation with Large Language and Vision Assistant enhanced training
- 6 September 2025
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 161
Personalized text-to-image generation with Large Language and Vision Assistant enhanced training
- Open Access
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
Recently, learning-based stereo matching networks have advanced significantly. However, they often lack robustness and struggle to achieve impressive cross-domain performance due to domain shifts and imbalanced disparity distributions among diverse datasets. Leveraging Vision Foundation Models (VFMs) can intuitively enhance the model's robustness, but integrating such a model into stereo matching cost-effectively to fully realize their robustness remains a key challenge. To address this, we propose SMoEStereo, a novel framework that adapts VFMs for stereo matching through a tailored, scene-specific fusion of Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) modules. SMoEStereo introduces MoE-LoRA with adaptive ranks and MoE-Adapter with adaptive kernel sizes. The former dynamically selects optimal experts within MoE to adapt varying scenes across domains, while the latter injects inductive bias into frozen VFMs to improve geometric feature extraction. Importantly, to mitigate computational overhead, we further propose a lightweight decision network that selectively activates MoE modules based on input complexity, balancing efficiency with accuracy. Extensive experiments demonstrate that our method exhibits state-of-the-art cross-domain and joint generalization across multiple benchmarks without dataset-specific adaptation. The code is available at \textcolor{red}{https://github.com/cocowy1/SMoE-Stereo}.
Open AccessLearning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
Recently, learning-based stereo matching networks have advanced significantly. However, they often lack robustness and struggle to achieve impressive cross-domain performance due to domain shifts and imbalanced disparity distributions among diverse datasets. Leveraging Vision Foundation Models (VFMs) can intuitively enhance the model's robustness, but integrating such a model into stereo matching cost-effectively to fully realize their robustness remains a key challenge. To address this, we propose SMoEStereo, a novel framework that adapts VFMs for stereo matching through a tailored, scene-specific fusion of Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) modules. SMoEStereo introduces MoE-LoRA with adaptive ranks and MoE-Adapter with adaptive kernel sizes. The former dynamically selects optimal experts within MoE to adapt varying scenes across domains, while the latter injects inductive bias into frozen VFMs to improve geometric feature extraction. Importantly, to mitigate computational overhead, we further propose a lightweight decision network that selectively activates MoE modules based on input complexity, balancing efficiency with accuracy. Extensive experiments demonstrate that our method exhibits state-of-the-art cross-domain and joint generalization across multiple benchmarks without dataset-specific adaptation. The code is available at \textcolor{red}{https://github.com/cocowy1/SMoE-Stereo}.
- 21 May 2025
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 156
VectorSketcher: Learning to create a vector-based free-hand sketch
- 21 May 2025
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 156
VectorSketcher: Learning to create a vector-based free-hand sketch