Zhihe Lu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhihe Lu has a total of 54 co-authors. Here are the top co-authors by number of shared publications:
- Xinchao Wang
National University of Singapore
6 shared publications - Zeyu Xiao
National University of Singapore, Singapore, Singapore
4 shared publications - Jun Cen
College of Computer Science and Technology, Zhejiang University, Hangzhou, China
3 shared publications - Yifan Ye
College of Science and Engineering, Hamad Bin Khalifa University, Education City, Doha, Qatar
3 shared publications - Da Li
Samsung AI Center Cambridge
2 shared publications - Jiaqi Ma
Computing and Mathematical Sciences Division, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
2 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 7089-7098
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 7089-7098
Open AccessDream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
- Open Access
- 15 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 7, 678-687
DepthPlay-Agent: Agentic Learning and Fusion for Depth Estimation
Monocular depth estimation in complex, dynamic environments remains challenging due to rapid object motion, texture repetitions, occlusions, and strong geometric constraints inherent in structured scenes. To address these challenges, we propose DepthPlay-Agent (DEAN), an agentic multi-model fusion framework that builds upon SOTA depth estimator backbones such as UNet++, HybridDepth, DINOv3, and ZoeDepth under a learnable Mixture-of-Experts controller. This controller embeds scene context through transformers and uses task priors to dynamically assign pe-pixel fusion weights. A domain-aware refinement module further enforces geometric consistency using planar and semantic segmentation cues. Beyond static fusion, DEAN introduces an agentic inference layer that dynamically regulates expert contributions and refinement strategies, enabling adaptive and interpretable decision-making. Experiments across four benchmarks (SoccerNet-Depth, KITTI, MPI-Sintel, NYU Depth V2) demonstrate that DEAN consistently achieves SOTA performance, improving AbsRel by up to 5% and reducing SILog by 8% over strong baselines. By coupling multi-modal intelligence with structured geometric reasoning, DEAN establishes a new paradigm for adaptive, context-aware depth estimation in dynamic real-world domains.
- 15 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of Signal Processing
- Vol. 7, 678-687
Open AccessDepthPlay-Agent: Agentic Learning and Fusion for Depth Estimation
Monocular depth estimation in complex, dynamic environments remains challenging due to rapid object motion, texture repetitions, occlusions, and strong geometric constraints inherent in structured scenes. To address these challenges, we propose DepthPlay-Agent (DEAN), an agentic multi-model fusion framework that builds upon SOTA depth estimator backbones such as UNet++, HybridDepth, DINOv3, and ZoeDepth under a learnable Mixture-of-Experts controller. This controller embeds scene context through transformers and uses task priors to dynamically assign pe-pixel fusion weights. A domain-aware refinement module further enforces geometric consistency using planar and semantic segmentation cues. Beyond static fusion, DEAN introduces an agentic inference layer that dynamically regulates expert contributions and refinement strategies, enabling adaptive and interpretable decision-making. Experiments across four benchmarks (SoccerNet-Depth, KITTI, MPI-Sintel, NYU Depth V2) demonstrate that DEAN consistently achieves SOTA performance, improving AbsRel by up to 5% and reducing SILog by 8% over strong baselines. By coupling multi-modal intelligence with structured geometric reasoning, DEAN establishes a new paradigm for adaptive, context-aware depth estimation in dynamic real-world domains.
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (8) , 9279-9286
Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models
Vision-Language-Action (VLA) models pretrained on large-scale multimodal datasets have emerged as powerful foundations for robotic perception and control. However, their massive scale, often billions of parameters, poses significant challenges for real-time deployment, as inference becomes computationally expensive and latency-sensitive in dynamic environments. To address this, we propose Token Expand-and-Merge-VLA (TEAM-VLA), a training-free token compression framework that accelerates VLA inference while preserving task performance. TEAM-VLA introduces a dynamic token expansion mechanism that identifies and samples additional informative tokens in the spatial vicinity of attention-highlighted regions, enhancing contextual completeness. These expanded tokens are then selectively merged in deeper layers under action-aware guidance, effectively reducing redundancy while maintaining semantic coherence. By coupling expansion and merging within a single feed-forward pass, TEAM-VLA achieves a balanced trade-off between efficiency and effectiveness, without any retraining or parameter updates. Extensive experiments on LIBERO benchmark demonstrate that TEAM-VLA consistently improves inference speed while maintaining or even surpassing the task success rate of full VLA models.
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (8) , 9279-9286
Token Expand-Merge: Training-Free Token Compression for Vision-Language-Action Models
Vision-Language-Action (VLA) models pretrained on large-scale multimodal datasets have emerged as powerful foundations for robotic perception and control. However, their massive scale, often billions of parameters, poses significant challenges for real-time deployment, as inference becomes computationally expensive and latency-sensitive in dynamic environments. To address this, we propose Token Expand-and-Merge-VLA (TEAM-VLA), a training-free token compression framework that accelerates VLA inference while preserving task performance. TEAM-VLA introduces a dynamic token expansion mechanism that identifies and samples additional informative tokens in the spatial vicinity of attention-highlighted regions, enhancing contextual completeness. These expanded tokens are then selectively merged in deeper layers under action-aware guidance, effectively reducing redundancy while maintaining semantic coherence. By coupling expansion and merging within a single feed-forward pass, TEAM-VLA achieves a balanced trade-off between efficiency and effectiveness, without any retraining or parameter updates. Extensive experiments on LIBERO benchmark demonstrate that TEAM-VLA consistently improves inference speed while maintaining or even surpassing the task success rate of full VLA models.
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (6) , 6967-6974
Self-Evolved Imitation Learning in Simulated World
Imitation learning has been a trend recently, yet training a generalist agent across multiple tasks still requires large-scale expert demonstrations, which are costly and labor-intensive to collect. To address the challenge of limited supervision, we propose Self-Evolved Imitation Learning (SEIL), a framework that progressively improves a few-shot model through simulator interactions. The model first attempts tasks in the simulator, from which successful trajectories are collected as new demonstrations for iterative refinement. To enhance the diversity of these demonstrations, SEIL employs dual-level augmentation: (i) Model-level, using an Exponential Moving Average (EMA) model to collaborate with the primary model, and (ii) Environment-level, introducing slight variations in initial object positions. We further introduce a lightweight selector that filters complementary and informative trajectories from the generated pool to ensure demonstration quality. These curated samples enable the model to achieve competitive performance with far fewer training examples. Extensive experiments on the LIBERO benchmark show that SEIL achieves a new state-of-the-art performance in few-shot imitation learning code is available at https://github.com/Jasper-aaa/SEIL.git.
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (6) , 6967-6974
Self-Evolved Imitation Learning in Simulated World
Imitation learning has been a trend recently, yet training a generalist agent across multiple tasks still requires large-scale expert demonstrations, which are costly and labor-intensive to collect. To address the challenge of limited supervision, we propose Self-Evolved Imitation Learning (SEIL), a framework that progressively improves a few-shot model through simulator interactions. The model first attempts tasks in the simulator, from which successful trajectories are collected as new demonstrations for iterative refinement. To enhance the diversity of these demonstrations, SEIL employs dual-level augmentation: (i) Model-level, using an Exponential Moving Average (EMA) model to collaborate with the primary model, and (ii) Environment-level, introducing slight variations in initial object positions. We further introduce a lightweight selector that filters complementary and informative trajectories from the generated pool to ensure demonstration quality. These curated samples enable the model to achieve competitive performance with far fewer training examples. Extensive experiments on the LIBERO benchmark show that SEIL achieves a new state-of-the-art performance in few-shot imitation learning code is available at https://github.com/Jasper-aaa/SEIL.git.
- Open Access
- 8 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Multi-scale Dynamic Network for Document Shadow Removal
- 8 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessMulti-scale Dynamic Network for Document Shadow Removal
- 19 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 6002-6015
COLA: Context-Aware Language-Driven Test-Time Adaptation
Test-time adaptation (TTA) has gained increasing popularity due to its efficacy in addressing “distribution shift” issue while simultaneously protecting data privacy. However, most prior methods assume that a paired source domain model and target domain sharing the same label space coexist, heavily limiting their applicability. In this paper, we investigate a more general source model capable of adaptation to multiple target domains without needing shared labels. This is achieved by using a pre-trained vision-language model (VLM), e.g., CLIP, that can recognize images through matching with class descriptions. While the zero-shot performance of VLMs is impressive, they struggle to effectively capture the distinctive attributes of a target domain. To that end, we propose a novel method – Context-aware Language-driven TTA (COLA). The proposed method incorporates a lightweight context-aware module that consists of three key components: a task-aware adapter, a context-aware unit, and a residual connection unit for exploring task-specific knowledge, domain-specific knowledge from the VLM and prior knowledge of the VLM, respectively. It is worth noting that the context-aware module can be seamlessly integrated into a frozen VLM, ensuring both minimal effort and parameter efficiency. Additionally, we introduce a Class-Balanced Pseudo-labeling (CBPL) strategy to mitigate the adverse effects caused by class imbalance. We demonstrate the effectiveness of our method not only in TTA scenarios but also in class generalisation tasks. The source code is available at https://github.com/NUDT-Bai-Group/COLA-TTA.
- 19 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 6002-6015
COLA: Context-Aware Language-Driven Test-Time Adaptation
Test-time adaptation (TTA) has gained increasing popularity due to its efficacy in addressing “distribution shift” issue while simultaneously protecting data privacy. However, most prior methods assume that a paired source domain model and target domain sharing the same label space coexist, heavily limiting their applicability. In this paper, we investigate a more general source model capable of adaptation to multiple target domains without needing shared labels. This is achieved by using a pre-trained vision-language model (VLM), e.g., CLIP, that can recognize images through matching with class descriptions. While the zero-shot performance of VLMs is impressive, they struggle to effectively capture the distinctive attributes of a target domain. To that end, we propose a novel method – Context-aware Language-driven TTA (COLA). The proposed method incorporates a lightweight context-aware module that consists of three key components: a task-aware adapter, a context-aware unit, and a residual connection unit for exploring task-specific knowledge, domain-specific knowledge from the VLM and prior knowledge of the VLM, respectively. It is worth noting that the context-aware module can be seamlessly integrated into a frozen VLM, ensuring both minimal effort and parameter efficiency. Additionally, we introduce a Class-Balanced Pseudo-labeling (CBPL) strategy to mitigate the adverse effects caused by class imbalance. We demonstrate the effectiveness of our method not only in TTA scenarios but also in class generalisation tasks. The source code is available at https://github.com/NUDT-Bai-Group/COLA-TTA.
- 14 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 1908-1920
Task-to-Instance Prompt Learning for Vision-Language Models at Test Time
Prompt learning has been recently introduced into the adaption of pre-trained vision-language models (VLMs) by tuning a set of trainable tokens to replace hand-crafted text templates. Despite the encouraging results achieved, existing methods largely rely on extra annotated data for training. In this paper, we investigate a more realistic scenario, where only the unlabeled test data is available. Existing test-time prompt learning methods often separately learn a prompt for each test sample. However, relying solely on a single sample heavily limits the performance of the learned prompts, as it neglects the task-level knowledge that can be gained from multiple samples. To that end, we propose a novel test-time prompt learning method of VLMs, called Task-to-Instance PromPt LEarning (TIPPLE), which adopts a two-stage training strategy to leverage both task- and instance-level knowledge. Specifically, we reformulate the effective online pseudo-labeling paradigm along with two tailored components: an auxiliary text classification task and a diversity regularization term, to serve the task-oriented prompt learning. After that, the learned task-level prompt is further combined with a tunable residual for each test sample to integrate with instance-level knowledge. We demonstrate the superior performance of TIPPLE on 15 downstream datasets, e.g., the average improvement of 1.87% over the state-of-the-art method, using ViT-B/16 visual backbone. Our code is open-sourced at https://github.com/zhiheLu/TIPPLE.
- 14 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 1908-1920
Task-to-Instance Prompt Learning for Vision-Language Models at Test Time
Prompt learning has been recently introduced into the adaption of pre-trained vision-language models (VLMs) by tuning a set of trainable tokens to replace hand-crafted text templates. Despite the encouraging results achieved, existing methods largely rely on extra annotated data for training. In this paper, we investigate a more realistic scenario, where only the unlabeled test data is available. Existing test-time prompt learning methods often separately learn a prompt for each test sample. However, relying solely on a single sample heavily limits the performance of the learned prompts, as it neglects the task-level knowledge that can be gained from multiple samples. To that end, we propose a novel test-time prompt learning method of VLMs, called Task-to-Instance PromPt LEarning (TIPPLE), which adopts a two-stage training strategy to leverage both task- and instance-level knowledge. Specifically, we reformulate the effective online pseudo-labeling paradigm along with two tailored components: an auxiliary text classification task and a diversity regularization term, to serve the task-oriented prompt learning. After that, the learned task-level prompt is further combined with a tunable residual for each test sample to integrate with instance-level knowledge. We demonstrate the superior performance of TIPPLE on 15 downstream datasets, e.g., the average improvement of 1.87% over the state-of-the-art method, using ViT-B/16 visual backbone. Our code is open-sourced at https://github.com/zhiheLu/TIPPLE.
- 24 November 2024
- Published by Springer Nature
- p. 346-363
Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach
- 24 November 2024
- Published by Springer Nature
- p. 346-363
Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach
- 9 November 2024
- Published by Springer Nature
- p. 342-359
Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-Time Adaptation
- 9 November 2024
- Published by Springer Nature
- p. 342-359
Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-Time Adaptation
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 3065-3074
Unraveling Motion Uncertainty for Local Motion Deblurring
In real-world photography, local motion blur often arises from the interplay between moving objects and stationary backgrounds during exposure. Existing deblurring methods face challenges in addressing local motion deblurring due to (i) the presence of arbitrary localized blurs and uncertain blur extents; (ii) the limited ability to accurately identify specific blurs resulting from ambiguous motion boundaries. These limitations often lead to suboptimal solutions when estimating blur maps and generating final deblurred images. To that end, we propose a novel method named Motion-Uncertainty-Guided Network (MUGNet), which harnesses a probabilistic representational model to explicitly address the intricacies stemming from motion uncertainties. Specifically, MUGNet consists of two key components, i.e., motion-uncertainty quantification (MUQ) module and motion-masked separable attention (M2SA) module, serving for complementary purposes. Concretely, MUQ aims to learn a conditional distribution for accurate and reliable blur map estimation, while the M2SA module is to enhance the representation of regions influenced by local motion blur and static background, which is achieved by promoting the establishment of extensive global interactions. We demonstrate the superiority of our MUGNet with extensive experiments. The code is publicly available at: https://github.com/zeyuxiao1997/MUGNet.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 3065-3074
Open AccessUnraveling Motion Uncertainty for Local Motion Deblurring
In real-world photography, local motion blur often arises from the interplay between moving objects and stationary backgrounds during exposure. Existing deblurring methods face challenges in addressing local motion deblurring due to (i) the presence of arbitrary localized blurs and uncertain blur extents; (ii) the limited ability to accurately identify specific blurs resulting from ambiguous motion boundaries. These limitations often lead to suboptimal solutions when estimating blur maps and generating final deblurred images. To that end, we propose a novel method named Motion-Uncertainty-Guided Network (MUGNet), which harnesses a probabilistic representational model to explicitly address the intricacies stemming from motion uncertainties. Specifically, MUGNet consists of two key components, i.e., motion-uncertainty quantification (MUQ) module and motion-masked separable attention (M2SA) module, serving for complementary purposes. Concretely, MUQ aims to learn a conditional distribution for accurate and reliable blur map estimation, while the M2SA module is to enhance the representation of regions influenced by local motion blur and static background, which is achieved by promoting the establishment of extensive global interactions. We demonstrate the superiority of our MUGNet with extensive experiments. The code is publicly available at: https://github.com/zeyuxiao1997/MUGNet.