Xu Zheng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Xu Zheng has a total of 42 co-authors. Here are the top co-authors by number of shared publications:
- Lin Wang
Nanyang technological university, Singapore, Singapore
6 shared publications - Yuanhuiyi Lyu
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China
3 shared publications - Daisuke Deguchi
Nagoya University, Nagoya, Japan
2 shared publications - Hiroshi Murase
Nagoya University, Nagoya, Japan
2 shared publications - Jialei Chen
Graduate School of Informatics, Nagoya University, Nagoya, Japan
2 shared publications - Jiazhou Zhou
Hong Kong University of Science and Technology
2 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 14061-14070
DIRE: Dual-grained Internal Refusal Estimation for VLM Jailbreak Detection
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 14061-14070
Open AccessDIRE: Dual-grained Internal Refusal Estimation for VLM Jailbreak Detection
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8509-8517
Guided Scale Decoupled Distillation
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8509-8517
Open AccessGuided Scale Decoupled Distillation
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 15530-15538
LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memory
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 15530-15538
Open AccessLiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memory
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 948-957
Beyond Localization: A Comprehensive Benchmark of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 948-957
Open AccessBeyond Localization: A Comprehensive Benchmark of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
- Open Access
- 23 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Event-based Lip Reading with Triplane Fusion Network
Lip reading via event cameras could capture subtle and similar lip motions with large dynamic range and microsecond latency, offering high temporal resolution over traditional frame-based cameras. However, existing approaches often overlook the exploitation of distinctive lip motion patterns, instead opting to adapt existing video recognition architectures. In this paper, we propose an event-specified framework named the Triplane Fusion Network (TF-Net) to harness lip movements through analysis from three distinct but complementary views. Specifically, while adhering to the standard XYT view, we further incorporate two additional perspectives: XT and YT, aiming to capitalize on the unique streaming nature of events over time. Given the three views, TF-Net contains multiple expert blocks for every view and mutual information exchange blocks that facilitate the multi-directional exchange of motion information among the different views. We observe that these three views complement each other, further enhancing the learning of the event-specified distribution. Extensive experiments validate the effectiveness of the proposed method, surpassing +1.6% and +2.3% accuracy over other competitive methods on the real-world dataset DVS-Lip and the synthetic dataset Modality, respectively.
- 23 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Open AccessEvent-based Lip Reading with Triplane Fusion Network
Lip reading via event cameras could capture subtle and similar lip motions with large dynamic range and microsecond latency, offering high temporal resolution over traditional frame-based cameras. However, existing approaches often overlook the exploitation of distinctive lip motion patterns, instead opting to adapt existing video recognition architectures. In this paper, we propose an event-specified framework named the Triplane Fusion Network (TF-Net) to harness lip movements through analysis from three distinct but complementary views. Specifically, while adhering to the standard XYT view, we further incorporate two additional perspectives: XT and YT, aiming to capitalize on the unique streaming nature of events over time. Given the three views, TF-Net contains multiple expert blocks for every view and mutual information exchange blocks that facilitate the multi-directional exchange of motion information among the different views. We observe that these three views complement each other, further enhancing the learning of the event-specified distribution. Extensive experiments validate the effectiveness of the proposed method, surpassing +1.6% and +2.3% accuracy over other competitive methods on the real-world dataset DVS-Lip and the synthetic dataset Modality, respectively.
- Open Access
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 1-12
BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 1-12
Open AccessBiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation
- Open Access
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-14
Omnidirectional Spatial Modeling from Correlated Panoramas
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-14
Open AccessOmnidirectional Spatial Modeling from Correlated Panoramas
- Open Access
- 1 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (3) , 3665-3678
CLIP-to-Seg Distillation for Zero-Shot Semantic Segmentation
CLIP has greatly advanced zero-shot segmentation by leveraging its strong visual-language association and generalization capability. However, directly adapting CLIP for segmentation often yields suboptimal results due to inconsistencies between image and pixel-level prediction objectives. Additionally, merely combining segmentation and CLIP models often leads to disjoint optimization, introducing significant computational overhead and additional parameters. To address these issues, we propose a novel CLIP-to-Seg Distillation approach, incorporating global and local distillation to flexibly transfer CLIP’s powerful zero-shot generalization capability to existing closed-set segmentation models. Global distillation leverages CLS tokens to condense segmentation features and distills high-level concepts to the segmentation model via image-level features. Local distillation adapts CLIP’s local semantic transferability to dense prediction tasks using object-level features, aided by pseudo-mask generation for latent class mining. To further generalize the CLIP-distilled segmentation model, we generate latent text embeddings for the mined latent classes by coordinating their text embeddings and dense features. Our method equips existing closed-set segmentation models with strong generalization capabilities for open concepts through effective and flexible CLIP-to-Seg distillation. Without relying on the CLIP model or introducing extra inference overhead, our method seamlessly integrates into existing closed-set segmentation models and enables zero-shot capability, achieving state-of-the-art performance on multiple benchmarks.
- 1 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (3) , 3665-3678
Open AccessCLIP-to-Seg Distillation for Zero-Shot Semantic Segmentation
CLIP has greatly advanced zero-shot segmentation by leveraging its strong visual-language association and generalization capability. However, directly adapting CLIP for segmentation often yields suboptimal results due to inconsistencies between image and pixel-level prediction objectives. Additionally, merely combining segmentation and CLIP models often leads to disjoint optimization, introducing significant computational overhead and additional parameters. To address these issues, we propose a novel CLIP-to-Seg Distillation approach, incorporating global and local distillation to flexibly transfer CLIP’s powerful zero-shot generalization capability to existing closed-set segmentation models. Global distillation leverages CLS tokens to condense segmentation features and distills high-level concepts to the segmentation model via image-level features. Local distillation adapts CLIP’s local semantic transferability to dense prediction tasks using object-level features, aided by pseudo-mask generation for latent class mining. To further generalize the CLIP-distilled segmentation model, we generate latent text embeddings for the mined latent classes by coordinating their text embeddings and dense features. Our method equips existing closed-set segmentation models with strong generalization capabilities for open concepts through effective and flexible CLIP-to-Seg distillation. Without relying on the CLIP model or introducing extra inference overhead, our method seamlessly integrates into existing closed-set segmentation models and enables zero-shot capability, achieving state-of-the-art performance on multiple benchmarks.
- 7 December 2024
- Published by Springer Nature
- p. 192-212
Centering the Value of Every Modality: Towards Efficient and Resilient Modality-Agnostic Semantic Segmentation
- 7 December 2024
- Published by Springer Nature
- p. 192-212
Centering the Value of Every Modality: Towards Efficient and Resilient Modality-Agnostic Semantic Segmentation
- 2 December 2024
- Published by Springer Nature
- p. 477-494
EventBind: Learning a Unified Representation to Bind Them All for Event-Based Open-World Understanding
- 2 December 2024
- Published by Springer Nature
- p. 477-494
EventBind: Learning a Unified Representation to Bind Them All for Event-Based Open-World Understanding