Bo Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Bo Zhang has a total of 64 co-authors. Here are the top co-authors by number of shared publications:
- Tao Chen
Fudan University
12 shared publications - Bin Wang
Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai, China
11 shared publications - Hancheng Ye
Duke University, Durham, NC, USA
5 shared publications - Jiayuan Fan
Academy for Engineering and Technology, Fudan University, Shanghai, China
5 shared publications - Baopu Li
Baidu, Sunnyvale, CA, USA
3 shared publications - Botian Shi
Shanghai Artificial Intelligence Laboratory
3 shared publications
Latest Publications
- Top Cited Papers
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (7) , 8044-8059
StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding
Charts are common in literature across various scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception that extracts information from the visual charts, or chart reasoning given the extracted data, e.g. in a tabular form. In this paper, we introduce StructChart, a novel framework that leverages Structured Triplet Representations (STR) to achieve a unified and label-efficient approach to chart perception and reasoning tasks, which is generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart data from the tubular form (linearized CSV) to STR, which can friendlily reduce the task gap between chart perception and reasoning. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the chart perception task performance. To augment the training, we further explore the potential of Large Language Models (LLMs) to enhance the diversity in both chart visual style and statistical information. Extensive experiments on various chart-related tasks demonstrate the effectiveness and potential of a unified chart perception-reasoning paradigm to push the frontier of chart understanding.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (7) , 8044-8059
Top Cited PapersStructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding
Charts are common in literature across various scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception that extracts information from the visual charts, or chart reasoning given the extracted data, e.g. in a tabular form. In this paper, we introduce StructChart, a novel framework that leverages Structured Triplet Representations (STR) to achieve a unified and label-efficient approach to chart perception and reasoning tasks, which is generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart data from the tubular form (linearized CSV) to STR, which can friendlily reduce the task gap between chart perception and reasoning. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the chart perception task performance. To augment the training, we further explore the potential of Large Language Models (LLMs) to enhance the diversity in both chart visual style and statistical information. Extensive experiments on various chart-related tasks demonstrate the effectiveness and potential of a unified chart perception-reasoning paradigm to push the frontier of chart understanding.
- Open Access
- 26 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18  (19391404) , 7207-7220
DSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images
Recently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture search (NAS) is capable of identifying the optimal network structure for multimodal RSIs and downstream tasks. However, due to the diverse spatial resolutions, complex channel dimensions, and drastic foreground scale variations of multimodal RSIs, challenges arise when employing NAS methods for precise classification: 1) Due to the complementary and redundant nature between different modalities in RSIs, determining the features within each modality for fusion becomes quite challenging; 2) The design of fusion operators does not take into account the spatial positions and channel relationships between different modalities of RSIs, making it difficult for the fused features to match downstream tasks. To address the these issues, we propose a Dual-Stage Feature Fusion framework based on NAS, termed DSF2-NAS, for the classification of multimodal RSIs. It primarily consists of two components: the Feature Candidate Search (FCS) module and the Fusion Operator Search (FOS) module, which execute sequentially. In the FCS module, a feature distance-based regularization approach is proposed to ensure fusion using multimodal features with the highest complementarity. Meanwhile, in the FOS module, a series of fusion operators are designed, which are based on spatial positions, channel relationships, and self-attention mechanisms, aiming to better integrate multimodal features with complex spatial and channel information. The proposed method has been evaluated on various datasets of multimodal RSIs, and experimental results consistently show that this method achieves state-of-the-art performance across multiple classification metrics. Our code is available at: https://github.com/I3ab/DSF2- NAS.
- 26 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18  (19391404) , 7207-7220
Open AccessDSF2-NAS: Dual-Stage Feature Fusion via Network Architecture Search for Classification of Multimodal Remote Sensing Images
Recently, the classification of multimodal remote sensing images (RSIs) has garnered significant attention due to its ability to provide rich information for various scenes on Earth. Compared to traditional feature fusion methods used for the classification of multimodal RSIs, neural architecture search (NAS) is capable of identifying the optimal network structure for multimodal RSIs and downstream tasks. However, due to the diverse spatial resolutions, complex channel dimensions, and drastic foreground scale variations of multimodal RSIs, challenges arise when employing NAS methods for precise classification: 1) Due to the complementary and redundant nature between different modalities in RSIs, determining the features within each modality for fusion becomes quite challenging; 2) The design of fusion operators does not take into account the spatial positions and channel relationships between different modalities of RSIs, making it difficult for the fused features to match downstream tasks. To address the these issues, we propose a Dual-Stage Feature Fusion framework based on NAS, termed DSF2-NAS, for the classification of multimodal RSIs. It primarily consists of two components: the Feature Candidate Search (FCS) module and the Fusion Operator Search (FOS) module, which execute sequentially. In the FCS module, a feature distance-based regularization approach is proposed to ensure fusion using multimodal features with the highest complementarity. Meanwhile, in the FOS module, a series of fusion operators are designed, which are based on spatial positions, channel relationships, and self-attention mechanisms, aiming to better integrate multimodal features with complex spatial and channel information. The proposed method has been evaluated on various datasets of multimodal RSIs, and experimental results consistently show that this method achieves state-of-the-art performance across multiple classification metrics. Our code is available at: https://github.com/I3ab/DSF2- NAS.
- 3 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47  (5) , 3657-3672
BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-Task Dense Predictions
Multi-task dense prediction aims at handling multiple pixel-wise prediction tasks within a unified network simultaneously for visual scene understanding. However, cross-task feature interactions of current methods are still suffering from incomplete levels of representations, less discriminative semantics in feature participants, and inefficient pair-wise task interaction processes. To tackle these under-explored issues, we propose a novel BridgeNet framework, which extracts comprehensive and discriminative intermediate Bridge Features, and conducts interactions based on them. Specifically, a Task Pattern Propagation (TPP) module is first applied to ensure highly semantic task-specific feature participants are prepared for subsequent interactions, and a Bridge Feature Extractor (BFE) is specially designed to selectively integrate both high-level and low-level representations to generate the comprehensive bridge features. Then, instead of conducting heavy pair-wise cross-task interactions, a Task-Feature Refiner (TFR) is developed to efficiently take guidance from bridge features and form final task predictions. To the best of our knowledge, this is the first work considering the completeness and quality of feature participants in cross-task interactions. Extensive experiments are conducted on NYUD-v2, Cityscapes and PASCAL Context benchmarks, and the superior performance shows the proposed architecture is effective and powerful in promoting different dense prediction tasks simultaneously.
- 3 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47  (5) , 3657-3672
BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-Task Dense Predictions
Multi-task dense prediction aims at handling multiple pixel-wise prediction tasks within a unified network simultaneously for visual scene understanding. However, cross-task feature interactions of current methods are still suffering from incomplete levels of representations, less discriminative semantics in feature participants, and inefficient pair-wise task interaction processes. To tackle these under-explored issues, we propose a novel BridgeNet framework, which extracts comprehensive and discriminative intermediate Bridge Features, and conducts interactions based on them. Specifically, a Task Pattern Propagation (TPP) module is first applied to ensure highly semantic task-specific feature participants are prepared for subsequent interactions, and a Bridge Feature Extractor (BFE) is specially designed to selectively integrate both high-level and low-level representations to generate the comprehensive bridge features. Then, instead of conducting heavy pair-wise cross-task interactions, a Task-Feature Refiner (TFR) is developed to efficiently take guidance from bridge features and form final task predictions. To the best of our knowledge, this is the first work considering the completeness and quality of feature participants in cross-task interactions. Extensive experiments are conducted on NYUD-v2, Cityscapes and PASCAL Context benchmarks, and the superior performance shows the proposed architecture is effective and powerful in promoting different dense prediction tasks simultaneously.
- 6 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63  (01962892) , 1-15
DF2RQ: Dynamic Feature Fusion via Region-Wise Queries for Semantic Segmentation of Multimodal Remote Sensing Data
Although remote sensing (RS) data with multiple modalities can be used to significantly improve the accuracy of semantic segmentation in RS data, how to effectively extract multimodal information through multimodal feature fusion remains a challenging task. Specifically, existing methods for multimodal feature fusion still face two major challenges: 1) Due to the diverse imaging mechanisms of multimodal RS data, the boundaries of the same foreground may vary across different modalities, leading to the inclusion of unwanted background semantics in the fused foreground features; 2) RS data from different modalities exhibit varying discriminative abilities for different foregrounds, making it challenging to determine the proportion of semantic information for each modality in the fusion results. To address the above issues, we propose a dynamic feature fusion method based on region-wise queries, namely DF 2 RQ, for SS of multimodal RS data. This method is primarily composed of two components: the spatial reconstruction (SR) module and the dynamic fusion (DF) module. Within the SR module, we propose a spatial reconstruction scheme that samples foreground features from different modalities, achieving independent reconstruction of different unimodal features, thereby alleviating the semantic mixing between foreground and background across modalities. In the DF module, a feature fusion scheme based on unimodal feature reference positions is proposed to obtain fusion weights for each modality, thereby enabling the dynamic fusion of complementary features from multiple modalities. The performance of the proposed method has been extensively evaluated on various multimodal RS datasets for SS, and the experimental results consistently show that the proposed method achieves state-of-the-art accuracy on multiple commonly used metrics. In addition, our code is available at https://github.com/I3ab/DF2RQ.
- 6 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63  (01962892) , 1-15
DF2RQ: Dynamic Feature Fusion via Region-Wise Queries for Semantic Segmentation of Multimodal Remote Sensing Data
Although remote sensing (RS) data with multiple modalities can be used to significantly improve the accuracy of semantic segmentation in RS data, how to effectively extract multimodal information through multimodal feature fusion remains a challenging task. Specifically, existing methods for multimodal feature fusion still face two major challenges: 1) Due to the diverse imaging mechanisms of multimodal RS data, the boundaries of the same foreground may vary across different modalities, leading to the inclusion of unwanted background semantics in the fused foreground features; 2) RS data from different modalities exhibit varying discriminative abilities for different foregrounds, making it challenging to determine the proportion of semantic information for each modality in the fusion results. To address the above issues, we propose a dynamic feature fusion method based on region-wise queries, namely DF 2 RQ, for SS of multimodal RS data. This method is primarily composed of two components: the spatial reconstruction (SR) module and the dynamic fusion (DF) module. Within the SR module, we propose a spatial reconstruction scheme that samples foreground features from different modalities, achieving independent reconstruction of different unimodal features, thereby alleviating the semantic mixing between foreground and background across modalities. In the DF module, a feature fusion scheme based on unimodal feature reference positions is proposed to obtain fusion weights for each modality, thereby enabling the dynamic fusion of complementary features from multiple modalities. The performance of the proposed method has been extensively evaluated on various multimodal RS datasets for SS, and the experimental results consistently show that the proposed method achieves state-of-the-art accuracy on multiple commonly used metrics. In addition, our code is available at https://github.com/I3ab/DF2RQ.
- Top Cited Papers
- 1 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34, 7436-7447
ChartX and ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously. However, their capacity to query information depicted in visual charts and engage in reasoning based on the queried contents remains under-explored. In this paper, to comprehensively and rigorously benchmark the ability of the off-the-shelf MLLMs in the chart domain, we construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data. Besides, we develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns, such as reasoning tasks in the field of charts or geometric images. We evaluate the chart-related ability of mainstream MLLMs and our ChartVLM on the proposed ChartX evaluation set. Extensive experiments demonstrate that ChartVLM surpasses both versatile and chart-related large models, including GPT-4V. We believe that our study can pave the way for further exploration in creating a more comprehensive chart evaluation set and developing more interpretable multi-modal models. Both ChartX and ChartVLM are available at: https://github.com/Alpha-Innovator/ChartVLM.
- 1 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34, 7436-7447
Top Cited PapersChartX and ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously. However, their capacity to query information depicted in visual charts and engage in reasoning based on the queried contents remains under-explored. In this paper, to comprehensively and rigorously benchmark the ability of the off-the-shelf MLLMs in the chart domain, we construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data. Besides, we develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns, such as reasoning tasks in the field of charts or geometric images. We evaluate the chart-related ability of mainstream MLLMs and our ChartVLM on the proposed ChartX evaluation set. Extensive experiments demonstrate that ChartVLM surpasses both versatile and chart-related large models, including GPT-4V. We believe that our study can pave the way for further exploration in creating a more comprehensive chart evaluation set and developing more interpretable multi-modal models. Both ChartX and ChartVLM are available at: https://github.com/Alpha-Innovator/ChartVLM.
- Open Access
- 27 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-9
On Reducing the Execution Latency of Superconducting Quantum Processors via Quantum Job Scheduling
- 27 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Open AccessOn Reducing the Execution Latency of Superconducting Quantum Processors via Quantum Job Scheduling
- 30 September 2024
- Published by Springer Nature
- p. 197-213
Reg-TTA3D: Better Regression Makes Better Test-Time Adaptive 3D Object Detection
- 30 September 2024
- Published by Springer Nature
- p. 197-213
Reg-TTA3D: Better Regression Makes Better Test-Time Adaptive 3D Object Detection
- 1 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (10) , 9078-9089
Few-Shot Cross-Domain Object Detection With Instance-Level Prototype-Based Meta-Learning
In typical unsupervised domain adaptive object detection, it is assumed that extensive unlabeled training data from the target domain can be easily obtained. However, in some access-constrained scenarios, massive target data cannot be guaranteed, but acquiring only a few target samples and annotating them may costs less. Therefore, inspired by the meta-learning success in few-shot tasks, we propose an Instance-level Prototype learning Network (IPNet) for solving the domain adaptive object detection under the supervised few-shot scenario in this work. To compensate for the target domain data deficiency, we fuse cropped instances from labeled images in both domains to learn a representative prototype for each class, by enforcing features of the same class’s instances but from different domains to be as close as possible. These prototypes are further employed to discriminate various features’ salience in an image, and separate foreground and background regions for respective domain alignment. Extensive experiments are conducted on several cross-domain scenarios, and their results show the consistent accuracy gains of the IPNet over state-of-the-art methods, e.g ., 10.4% mAP increase on Cityscapes-to-FoggyCityscapes setting and 3.0% mAP increase on Sim10k-to-Cityscapes setting.
- 1 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (10) , 9078-9089
Few-Shot Cross-Domain Object Detection With Instance-Level Prototype-Based Meta-Learning
In typical unsupervised domain adaptive object detection, it is assumed that extensive unlabeled training data from the target domain can be easily obtained. However, in some access-constrained scenarios, massive target data cannot be guaranteed, but acquiring only a few target samples and annotating them may costs less. Therefore, inspired by the meta-learning success in few-shot tasks, we propose an Instance-level Prototype learning Network (IPNet) for solving the domain adaptive object detection under the supervised few-shot scenario in this work. To compensate for the target domain data deficiency, we fuse cropped instances from labeled images in both domains to learn a representative prototype for each class, by enforcing features of the same class’s instances but from different domains to be as close as possible. These prototypes are further employed to discriminate various features’ salience in an image, and separate foreground and background regions for respective domain alignment. Extensive experiments are conducted on several cross-domain scenarios, and their results show the consistent accuracy gains of the IPNet over state-of-the-art methods, e.g ., 10.4% mAP increase on Cityscapes-to-FoggyCityscapes setting and 3.0% mAP increase on Sim10k-to-Cityscapes setting.
- 28 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (8) , 7165-7175
Push-and-Pull: A General Training Framework With Differential Augmentor for Domain Generalized Point Cloud Classification
As a fundamental task of 3D perception, point cloud recognition has shown significant progress in recent years. However, existing methods still face challenges when dealing with geometry differences, resulting in performance degradation when a distribution gap exists between the training and testing data, also known as domain generalization. In this work, we focus on this problem and propose a general training framework, named Push-and-Pull, aimed at effectively improving the generalization ability of models on unseen target domains. Specifically, our framework first introduces a learnable 3D data augmentor to generate new training point clouds, which helps to reduce the domain bias and enrich the source training set. Also, an adversarial training strategy is proposed to push the augmented samples away from the original ones in the latent space and meanwhile keep the geometric structure. Second, based on the original and augmented samples, a dual-level consistency regularization strategy on logits and feature spaces is designed to pull the deviated representations back to their original space as close as possible, and promote discriminative and domain-agnostic representations. These two steps are iteratively optimized to enhance the overall performance. Extensive experiments on the PointDA-10 and Sim2Real benchmarks consistently demonstrate the effectiveness of our proposed framework.
- 28 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (8) , 7165-7175
Push-and-Pull: A General Training Framework With Differential Augmentor for Domain Generalized Point Cloud Classification
As a fundamental task of 3D perception, point cloud recognition has shown significant progress in recent years. However, existing methods still face challenges when dealing with geometry differences, resulting in performance degradation when a distribution gap exists between the training and testing data, also known as domain generalization. In this work, we focus on this problem and propose a general training framework, named Push-and-Pull, aimed at effectively improving the generalization ability of models on unseen target domains. Specifically, our framework first introduces a learnable 3D data augmentor to generate new training point clouds, which helps to reduce the domain bias and enrich the source training set. Also, an adversarial training strategy is proposed to push the augmented samples away from the original ones in the latent space and meanwhile keep the geometric structure. Second, based on the original and augmented samples, a dual-level consistency regularization strategy on logits and feature spaces is designed to pull the deviated representations back to their original space as close as possible, and promote discriminative and domain-agnostic representations. These two steps are iteratively optimized to enhance the overall performance. Extensive experiments on the PointDA-10 and Sim2Real benchmarks consistently demonstrate the effectiveness of our proposed framework.
- 15 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (7) , 5951-5962
Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?
Point cloud based 3D deep model has wide applications in many applications such as autonomous driving, house robot, etc. Inspired by the recent prompt learning in natural language processing, this work proposes a novel Multi-view Vision Fusion Network (MvNet) for few-shot 3D point cloud classification. MvNet investigates the possibility of leveraging the off-the-shelf 2D pre-trained models to achieve the few-shot classification, which can alleviate the over-dependence issue of the existing baseline models towards the large-scale annotated 3D point cloud data. Specifically, MvNet first encodes a 3D point cloud into multi-view image features for a number of different views. Then, a novel multi-view prompt fusion module is developed to fuse information from different views effectively to bridge the gap between 3D point cloud data and 2D pre-trained models. A set of 2D image prompts can then be derived to better describe the suitable prior knowledge for a large-scale pre-trained image model for few-shot 3D point cloud classification. Extensive experiments on ModelNet, ScanObjectNN, and ShapeNet datasets demonstrate that MvNet achieves new state-of-the-art performance for 3D few-shot point cloud image classification. The source code of this work is available at https://github.com/invictus717/MetaTransformer.
- 15 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (7) , 5951-5962
Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?
Point cloud based 3D deep model has wide applications in many applications such as autonomous driving, house robot, etc. Inspired by the recent prompt learning in natural language processing, this work proposes a novel Multi-view Vision Fusion Network (MvNet) for few-shot 3D point cloud classification. MvNet investigates the possibility of leveraging the off-the-shelf 2D pre-trained models to achieve the few-shot classification, which can alleviate the over-dependence issue of the existing baseline models towards the large-scale annotated 3D point cloud data. Specifically, MvNet first encodes a 3D point cloud into multi-view image features for a number of different views. Then, a novel multi-view prompt fusion module is developed to fuse information from different views effectively to bridge the gap between 3D point cloud data and 2D pre-trained models. A set of 2D image prompts can then be derived to better describe the suitable prior knowledge for a large-scale pre-trained image model for few-shot 3D point cloud classification. Extensive experiments on ModelNet, ScanObjectNN, and ShapeNet datasets demonstrate that MvNet achieves new state-of-the-art performance for 3D few-shot point cloud image classification. The source code of this work is available at https://github.com/invictus717/MetaTransformer.