Hongyuan Zhu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Hongyuan Zhu has a total of 161 co-authors. Here are the top co-authors by number of shared publications:
- Xi Peng
College of Computer Science, Sichuan University, Chengdu, China
12 shared publications - Peng Hu
College of Computer Science, Sichuan University, Chengdu, China
8 shared publications - Tao Chen
Fudan University
7 shared publications - Shijian Lu
Nanyang Technological University
6 shared publications - Dezhong Peng
Sichuan University, Chengdu, China
5 shared publications - Hanwang Zhang
Nanyang Technological University
5 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 11384-11393
SARNet: Rethinking Correspondence Granularity for Cross-Domain Few-Shot Segmentation
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 11384-11393
Open AccessSARNet: Rethinking Correspondence Granularity for Cross-Domain Few-Shot Segmentation
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 18040-18041
HUMA'26: The 6th International Workshop on Human-centric Multimedia Analysis Summary
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 18040-18041
Open AccessHUMA'26: The 6th International Workshop on Human-centric Multimedia Analysis Summary
- 16 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. PP, 1-15
Heterogeneous Multimodal Multilabel Behavioral Context Recognition Using Wearable Sensors and Mobile Devices
Accurate behavioral context recognition (BCR) is essential for advancing social intelligence in smart home settings and social companion pairing. With advancements in mobile computing technologies, wearable sensors, and mobile devices have become crucial for multimodal data collection to interpret behavioral context. However, existing methods may overlook the multilabel attribute of behavioral context, limiting their ability to model modality-to-label and label-to-label dependencies. Moreover, the unique label heterogeneity in BCR, arising from category-specific differences among labels, is often underestimated. To overcome these challenges, we propose a heterogeneous multimodal multilabel recognition method to capture the relationships between modalities and labels, while taking the label heterogeneity into account. Using specialized heterogeneous decoders with self-adaptive attention and a heterogeneous graph attention network based on the dual-level attention mechanism, our method captures complex dependencies among labels and modalities. Experiments on two benchmark datasets demonstrate the superior performance of our approach, with ablation studies validating the contributions of each component.
- 16 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. PP, 1-15
Heterogeneous Multimodal Multilabel Behavioral Context Recognition Using Wearable Sensors and Mobile Devices
Accurate behavioral context recognition (BCR) is essential for advancing social intelligence in smart home settings and social companion pairing. With advancements in mobile computing technologies, wearable sensors, and mobile devices have become crucial for multimodal data collection to interpret behavioral context. However, existing methods may overlook the multilabel attribute of behavioral context, limiting their ability to model modality-to-label and label-to-label dependencies. Moreover, the unique label heterogeneity in BCR, arising from category-specific differences among labels, is often underestimated. To overcome these challenges, we propose a heterogeneous multimodal multilabel recognition method to capture the relationships between modalities and labels, while taking the label heterogeneity into account. Using specialized heterogeneous decoders with self-adaptive attention and a heterogeneous graph attention network based on the dual-level attention mechanism, our method captures complex dependencies among labels and modalities. Experiments on two benchmark datasets demonstrate the superior performance of our approach, with ablation studies validating the contributions of each component.
- 27 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-13
Rendered 2D Semantic and Generative Priors Guided 3D Multi-Object Grounding
3D multi-object visual grounding aims to identify and localize all objects in a 3D scene that correspond to a given text description. Unlike traditional single-object grounding, this task presents additional challenges as point clouds inherently lack fine-grained details, making it difficult to capture subtle object features and contextual information. Moreover, textual descriptions are inherently limited in perceiving and understanding complex 3D environments, especially in scenarios with high object similarity or intricate spatial arrangements. To tackle the above challenges, we propose SGMG, a Rendered 2D Semantic and Generative priors guided 3D Multi-object Grounding Framework. The SGMG framework introduces two key innovations that work cohesively to enhance grounding accuracy. First, a Generative-Assistant(GA) Module leverages the capabilities of a generative model to provide enriched scene prior information and capture the fine-grained scene details. Second, the Semantic-Augment Fusion(SAF) Module is designed to improve the representation of text features from the vision features, thereby boosting the accuracy of multimodal information interactions. Furthermore, we introduce a multi-level fusion mechanism, ensuring that semantic and spatial relationships between objects are preserved and effectively leveraged during the grounding process. Experimental results demonstrate that SGMG achieves state-of-the-art performance in multi-object 3D grounding and competitive results in traditional single-object tasks, highlighting its effectiveness in diverse scenarios.
- 27 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-13
Rendered 2D Semantic and Generative Priors Guided 3D Multi-Object Grounding
3D multi-object visual grounding aims to identify and localize all objects in a 3D scene that correspond to a given text description. Unlike traditional single-object grounding, this task presents additional challenges as point clouds inherently lack fine-grained details, making it difficult to capture subtle object features and contextual information. Moreover, textual descriptions are inherently limited in perceiving and understanding complex 3D environments, especially in scenarios with high object similarity or intricate spatial arrangements. To tackle the above challenges, we propose SGMG, a Rendered 2D Semantic and Generative priors guided 3D Multi-object Grounding Framework. The SGMG framework introduces two key innovations that work cohesively to enhance grounding accuracy. First, a Generative-Assistant(GA) Module leverages the capabilities of a generative model to provide enriched scene prior information and capture the fine-grained scene details. Second, the Semantic-Augment Fusion(SAF) Module is designed to improve the representation of text features from the vision features, thereby boosting the accuracy of multimodal information interactions. Furthermore, we introduce a multi-level fusion mechanism, ensuring that semantic and spatial relationships between objects are preserved and effectively leveraged during the grounding process. Experimental results demonstrate that SGMG achieves state-of-the-art performance in multi-object 3D grounding and competitive results in traditional single-object tasks, highlighting its effectiveness in diverse scenarios.
- Open Access
Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover's Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.
Open AccessBeyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However, this assumption breaks down for non-iconic data, where different views may contain distinct objects or semantic information. In this paper, we investigate the effectiveness of SSL when instance consistency is not guaranteed. Through extensive ablation studies, we demonstrate that SSL can still learn meaningful representations even when positive pairs lack strict instance consistency. Furthermore, our analysis further reveals that increasing view diversity, by enforcing zero overlapping or using smaller crop scales, can enhance downstream performance on classification and dense prediction tasks. However, excessive diversity is found to reduce effectiveness, suggesting an optimal range for view diversity. To quantify this, we adopt the Earth Mover's Distance (EMD) as an estimator to measure mutual information between views, finding that moderate EMD values correlate with improved SSL learning, providing insights for future SSL framework design. We validate our findings across a range of settings, highlighting their robustness and applicability on diverse data sources.
- 23 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6908-6921
SF-City: A Source-Free Domain Adaptation Method for City-Scale Point Cloud Semantic Segmentation
City-scale point cloud semantic segmentation is an important yet challenging task. Despite progress, existing methods rely heavily on point-wise annotations. An alternative solution is to apply the Unsupervised Domain Adaptation (UDA) approach. Recently, the 2D foundation model has achieved significant progress with training with internet-scale images. Therefore, adapting 2D foundation models to 3D City-scale point clouds is an attempting idea. Due to the data protection and storage issue, 2D source domain data is typically unavailable. Thus, we focus on Source-Free Domain Adaptation (SFDA) and propose a Source-Free City-scale point cloud semantic segmentation method, namely SF-City. Our method leverages knowledge from 2D pre-trained models to generate point-wise pseudo labels for training a 3D semantic segmentation network. We convert point clouds into remote-sensing-like images using Bird's-Eye-View (BEV) projection. However, directly using source models for pseudo label generation is hindered by domain gaps such as viewpoint variations, concept divergences, and geometry loss. To tackle these problems, we propose a Multi-scale Content Feature Extractor (MCFE) to extract holistic and contextual feature representations. Then, an Uncertainty-guided Inter-Model Feature Integrator (UIFI) is introduced to integrate inherent knowledge across source models. Furthermore, the Geometric-guided Pseudo Label Generator (GPLG) is leveraged to introduce geometric information to regulate pseudo labels. Through extensive experiments on two public benchmarks, SF-City demonstrates superior performance, achieving an mIoU of 28.8% on the SensatUrban dataset, outperforming recent state-of-the-art methods CLIPFO3D by about 6.3%.
- 23 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6908-6921
SF-City: A Source-Free Domain Adaptation Method for City-Scale Point Cloud Semantic Segmentation
City-scale point cloud semantic segmentation is an important yet challenging task. Despite progress, existing methods rely heavily on point-wise annotations. An alternative solution is to apply the Unsupervised Domain Adaptation (UDA) approach. Recently, the 2D foundation model has achieved significant progress with training with internet-scale images. Therefore, adapting 2D foundation models to 3D City-scale point clouds is an attempting idea. Due to the data protection and storage issue, 2D source domain data is typically unavailable. Thus, we focus on Source-Free Domain Adaptation (SFDA) and propose a Source-Free City-scale point cloud semantic segmentation method, namely SF-City. Our method leverages knowledge from 2D pre-trained models to generate point-wise pseudo labels for training a 3D semantic segmentation network. We convert point clouds into remote-sensing-like images using Bird's-Eye-View (BEV) projection. However, directly using source models for pseudo label generation is hindered by domain gaps such as viewpoint variations, concept divergences, and geometry loss. To tackle these problems, we propose a Multi-scale Content Feature Extractor (MCFE) to extract holistic and contextual feature representations. Then, an Uncertainty-guided Inter-Model Feature Integrator (UIFI) is introduced to integrate inherent knowledge across source models. Furthermore, the Geometric-guided Pseudo Label Generator (GPLG) is leveraged to introduce geometric information to regulate pseudo labels. Through extensive experiments on two public benchmarks, SF-City demonstrates superior performance, achieving an mIoU of 28.8% on the SensatUrban dataset, outperforming recent state-of-the-art methods CLIPFO3D by about 6.3%.
- 22 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6986-6995
PointCloud-Text Matching: Benchmark Dataset and Baseline
In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching (PTM), which aims to identify the exact cross-modal instance that matches a given point-cloud query or text query. PTM has potential applications in various scenarios, such as indoor/urban-canyon localization and scene retrieval. However, there is a lack of suitable and targeted datasets for PTM in practice. To address this issue, we present a new PTM benchmark dataset, namely SceneDepict-3D2T. We observe that the data poses significant challenges due to its inherent characteristics, such as the sparsity, noise, or disorder of point clouds and the ambiguity, vagueness, or incompleteness of texts, which render existing cross-modal matching methods ineffective for PTM. To overcome these challenges, we propose a PTM baseline, named Robust PointCloud-Text Matching method (RoMa). RoMa consists of two key modules: a Dual Attention Perception module (DAP) and a Robust Negative Contrastive Learning module (RNCL). Specifically, DAP leverages token-level and feature-level attention mechanisms to adaptively focus on useful local and global features, and aggregate them into common representations, thereby reducing the adverse impact of noise and ambiguity. To handle noisy correspondence, RNCL enhances robustness against mismatching by dividing negative pairs into clean and noisy subsets and assigning them forward and reverse optimization directions, respectively. We conduct extensive experiments on our benchmarks and demonstrate the superiority of our RoMa.
- 22 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6986-6995
PointCloud-Text Matching: Benchmark Dataset and Baseline
In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching (PTM), which aims to identify the exact cross-modal instance that matches a given point-cloud query or text query. PTM has potential applications in various scenarios, such as indoor/urban-canyon localization and scene retrieval. However, there is a lack of suitable and targeted datasets for PTM in practice. To address this issue, we present a new PTM benchmark dataset, namely SceneDepict-3D2T. We observe that the data poses significant challenges due to its inherent characteristics, such as the sparsity, noise, or disorder of point clouds and the ambiguity, vagueness, or incompleteness of texts, which render existing cross-modal matching methods ineffective for PTM. To overcome these challenges, we propose a PTM baseline, named Robust PointCloud-Text Matching method (RoMa). RoMa consists of two key modules: a Dual Attention Perception module (DAP) and a Robust Negative Contrastive Learning module (RNCL). Specifically, DAP leverages token-level and feature-level attention mechanisms to adaptively focus on useful local and global features, and aggregate them into common representations, thereby reducing the adverse impact of noise and ambiguity. To handle noisy correspondence, RNCL enhances robustness against mismatching by dividing negative pairs into clean and noisy subsets and assigning them forward and reverse optimization directions, respectively. We conduct extensive experiments on our benchmarks and demonstrate the superiority of our RoMa.
- 20 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6273-6283
WI3D: Weakly Incremental 3D Detection via Vision Foundation Models
Class-incremental 3D object detection demands a 3D detector to locate and recognize novel categories in a stream fashion while preserving its base detection ability. However, existing methods require delicate 3D annotations for learning novel categories, resulting in significant labeling costs. To this end, we explore a label-efficient approach called Weakly Incremental 3D Detection (WI3D), which teaches a 3D detector to learn incrementally with off-the-shelf vision foundation models. We propose a novel dual-teaching framework incorporating both intra-modal and inter-modal knowledge from pseudo labels and feature space. Specifically, our framework features a class-agnostic pseudo-label refinement module, designed for the generation of high-quality 3D pseudo labels. This module is built on a lightweight transformer that models the spatial relationships between pseudo labels and their interactions with rich contextual information in point clouds. Additionally, we introduce a cross-modal knowledge transfer module to enhance the representation learning of novel classes, along with a reweighting knowledge distillation strategy that dynamically assesses and distills knowledge from previously learned categories. Extensive experiments show that our approach can efficiently learn novel concepts while preserving knowledge of base classes in WI3D scenarios, and surpass baseline approaches on both SUN-RGBD and ScanNet.
- 20 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6273-6283
WI3D: Weakly Incremental 3D Detection via Vision Foundation Models
Class-incremental 3D object detection demands a 3D detector to locate and recognize novel categories in a stream fashion while preserving its base detection ability. However, existing methods require delicate 3D annotations for learning novel categories, resulting in significant labeling costs. To this end, we explore a label-efficient approach called Weakly Incremental 3D Detection (WI3D), which teaches a 3D detector to learn incrementally with off-the-shelf vision foundation models. We propose a novel dual-teaching framework incorporating both intra-modal and inter-modal knowledge from pseudo labels and feature space. Specifically, our framework features a class-agnostic pseudo-label refinement module, designed for the generation of high-quality 3D pseudo labels. This module is built on a lightweight transformer that models the spatial relationships between pseudo labels and their interactions with rich contextual information in point clouds. Additionally, we introduce a cross-modal knowledge transfer module to enhance the representation learning of novel classes, along with a reweighting knowledge distillation strategy that dynamically assesses and distills knowledge from previously learned categories. Extensive experiments show that our approach can efficiently learn novel concepts while preserving knowledge of base classes in WI3D scenarios, and surpass baseline approaches on both SUN-RGBD and ScanNet.
- Open Access
- 5 May 2025
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (8) , 5872-5881
Correction: Consistent Prompt Tuning for Generalized Category Discovery
- 5 May 2025
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (8) , 5872-5881
Open AccessCorrection: Consistent Prompt Tuning for Generalized Category Discovery
- 1 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 2228-2240
Object Adaptive Self-Supervised Dense Visual Pre-Training
Self-supervised visual pre-training models have achieved significant success without employing expensive annotations. Nevertheless, most of these models focus on iconic single-instance datasets (e.g. ImageNet), ignoring the insufficient discriminative representation for non-iconic multi-instance datasets (e.g. COCO). In this paper, we propose a novel Object Adaptive Dense Pre-training (OADP) method to learn the visual representation directly on the multi-instance datasets (e.g., PASCAL VOC and COCO) for dense prediction tasks (e.g., object detection and instance segmentation). We present a novel object-aware and learning-adaptive random view augmentation to focus the contrastive learning to enhance the discrimination of object presentations from large to small scale during different learning stages. Furthermore, the representations across different scale and resolutions are integrated so that the method can learn diverse representations. In the experiment, we evaluated OADP pre-trained on PASCAL VOC and COCO. Results show that our method has better performances than most existing state-of-the-art methods when transferring to various downstream tasks, including image classification, object detection, instance segmentation and semantic segmentation.
- 1 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 2228-2240
Object Adaptive Self-Supervised Dense Visual Pre-Training
Self-supervised visual pre-training models have achieved significant success without employing expensive annotations. Nevertheless, most of these models focus on iconic single-instance datasets (e.g. ImageNet), ignoring the insufficient discriminative representation for non-iconic multi-instance datasets (e.g. COCO). In this paper, we propose a novel Object Adaptive Dense Pre-training (OADP) method to learn the visual representation directly on the multi-instance datasets (e.g., PASCAL VOC and COCO) for dense prediction tasks (e.g., object detection and instance segmentation). We present a novel object-aware and learning-adaptive random view augmentation to focus the contrastive learning to enhance the discrimination of object presentations from large to small scale during different learning stages. Furthermore, the representations across different scale and resolutions are integrated so that the method can learn diverse representations. In the experiment, we evaluated OADP pre-trained on PASCAL VOC and COCO. Results show that our method has better performances than most existing state-of-the-art methods when transferring to various downstream tasks, including image classification, object detection, instance segmentation and semantic segmentation.