Shaowei Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Shaowei Wang has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Jun Liu
Xi'an Jiaotong University, Xi'an, China
5 shared publications - Lingling Zhang
School of Computer Science and Technology Xi'an Jiaotong University
5 shared publications - Xinyu Zhang
School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an, Shaanxi, China
3 shared publications Qianying Wang
2 shared publicationsQinghua Zheng
Xi An Jiao Tong Univ, Dept Comp Sci, Xian 710049, Shaanxi, Peoples R China
2 shared publications- Tao Qin
School of Computer Science and Technology, Xian Jiaotong University, Xian, China
2 shared publications
Latest Publications
- 26 December 2025
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 264
Memory-enriched thought-by-thought framework for complex Diagram Question Answering
- 26 December 2025
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 264
Memory-enriched thought-by-thought framework for complex Diagram Question Answering
- 23 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 2141-2154
Alignment-Guided Self-Supervised Learning for Diagram Question Answering
Diagram question answering (DQA), which is defined as answering natural language questions according to the visual diagram context, has attracted attention and has recently become a new benchmark for evaluating the complex reasoning ability of models. However, this reasoning task is extremely challenging because of the inclusion of abstract visual objects and specialized textual terms, as well as the complex relationships between them. The rarity of data caused by the high cost of annotation also makes large-scale deep models invalid for the DQA task. To address the above challenges, this paper proposes the cross-modal alignment-guided self-supervised learning model for DQA (CAS-DQA). Unlike previous works, the CAS-DQA model focuses on learning internal visual-textual object relationships, innovatively proposes an attention mechanism module based on object alignment, and effectively integrates cross-modal knowledge units for diagram understanding. In addition, the CAS-DQA model constructs two self-supervised learning (SSL) tasks via intermediate results of visual-textual object alignment. These two tasks exploit the unnoticed objects inside the diagram to fully and completely understand the diagram. They also effectively increase the amount of diagram question-answering data to address the challenge of data scarcity. To the best of our knowledge, the CAS-DQA model is the first to extend SSL strategies to the diagram question-answering task. We evaluate the CAS-DQA model on three different datasets. The results of extensive experiments show that our model significantly outperforms baselines on different scenarios and that the internal object alignment module and self-supervised tasks produce excellent results. The core code is available at https://github.com/WayneWong97/CAS-DQA .
- 23 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 2141-2154
Alignment-Guided Self-Supervised Learning for Diagram Question Answering
Diagram question answering (DQA), which is defined as answering natural language questions according to the visual diagram context, has attracted attention and has recently become a new benchmark for evaluating the complex reasoning ability of models. However, this reasoning task is extremely challenging because of the inclusion of abstract visual objects and specialized textual terms, as well as the complex relationships between them. The rarity of data caused by the high cost of annotation also makes large-scale deep models invalid for the DQA task. To address the above challenges, this paper proposes the cross-modal alignment-guided self-supervised learning model for DQA (CAS-DQA). Unlike previous works, the CAS-DQA model focuses on learning internal visual-textual object relationships, innovatively proposes an attention mechanism module based on object alignment, and effectively integrates cross-modal knowledge units for diagram understanding. In addition, the CAS-DQA model constructs two self-supervised learning (SSL) tasks via intermediate results of visual-textual object alignment. These two tasks exploit the unnoticed objects inside the diagram to fully and completely understand the diagram. They also effectively increase the amount of diagram question-answering data to address the challenge of data scarcity. To the best of our knowledge, the CAS-DQA model is the first to extend SSL strategies to the diagram question-answering task. We evaluate the CAS-DQA model on three different datasets. The results of extensive experiments show that our model significantly outperforms baselines on different scenarios and that the internal object alignment module and self-supervised tasks produce excellent results. The core code is available at https://github.com/WayneWong97/CAS-DQA .
- 16 April 2024
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 244
Multi-view cognition with path search for one-shot part labeling
- 16 April 2024
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 244
Multi-view cognition with path search for one-shot part labeling
- 13 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 2131-2144
Alignment Relation is What You Need for Diagram Parsing
As a knowledge carrier, the diagram is widely distributed in many aspects of human life, such as textbooks, architectural drawings, and documents. Different from natural images, representations of visual elements in the diagram are sparser, and similar visual representations can reflect dissimilar semantics. Thus, current methods fail to capture the visual elements with precise semantics. To address this issue, regarding the aligned visual and textual elements as pairs is the way to assign the precise semantics of textual elements to visual elements. We build the first diagram dataset named align diagram element (ADE), which includes annotations for alignment relations between visual and textual elements. And we propose a visual-textual alignment model (VTAM) including graph construction and optimal aligning phases. In the graph construction phase, the relational graphs are constructed between different elements with four relational operators. The relational operators are designed to measure the relations between different elements, according to distance, connection line, inclusion, and feature similarity. In the optimal aligning phase, the representation at each visual-textual pair is improved as a weighted sum of the representations on all relational graphs. Experimental results show that our VTAM achieves a significant improvement of 10.9% on mean test folds of the ADE dataset than the current best competitor. In order to explore the role of alignment relations in diagram parsing, we introduce VTAM to diagram-related tasks, such as diagram question answering (DQA). And we achieve 2.8% to 5.9% and 4.6% to 5.1% improvements on AI2D and Foodwebs after adding VTAM. Our dataset and code are released at: https://github.com/ADE-dataset/ADE-dataset.
- 13 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 2131-2144
Alignment Relation is What You Need for Diagram Parsing
As a knowledge carrier, the diagram is widely distributed in many aspects of human life, such as textbooks, architectural drawings, and documents. Different from natural images, representations of visual elements in the diagram are sparser, and similar visual representations can reflect dissimilar semantics. Thus, current methods fail to capture the visual elements with precise semantics. To address this issue, regarding the aligned visual and textual elements as pairs is the way to assign the precise semantics of textual elements to visual elements. We build the first diagram dataset named align diagram element (ADE), which includes annotations for alignment relations between visual and textual elements. And we propose a visual-textual alignment model (VTAM) including graph construction and optimal aligning phases. In the graph construction phase, the relational graphs are constructed between different elements with four relational operators. The relational operators are designed to measure the relations between different elements, according to distance, connection line, inclusion, and feature similarity. In the optimal aligning phase, the representation at each visual-textual pair is improved as a weighted sum of the representations on all relational graphs. Experimental results show that our VTAM achieves a significant improvement of 10.9% on mean test folds of the ADE dataset than the current best competitor. In order to explore the role of alignment relations in diagram parsing, we introduce VTAM to diagram-related tasks, such as diagram question answering (DQA). And we achieve 2.8% to 5.9% and 4.6% to 5.1% improvements on AI2D and Foodwebs after adding VTAM. Our dataset and code are released at: https://github.com/ADE-dataset/ADE-dataset.
- Open Access
- 30 July 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 16 (6) , 1-20
Computer Science Diagram Understanding with Topology Parsing
Diagram is a special form of visual expression for representing complex concepts, logic, and knowledge, which widely appears in educational scenes such as textbooks, blogs, and encyclopedias. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. In this article, we construct the first novel geometric type of diagrams dataset in Computer Science field, which has more abstract expressions and complex logical relations. The dataset has exhaustive annotations of objects and relations for about 1,300 diagrams and 3,500 question-answer pairs. We introduce the tasks of diagram classification (DC) and diagram question answering (DQA) based on the new dataset, and propose the Diagram Paring Net (DPN) that focuses on analyzing the topological structure and text information of diagrams. We use DPN-based models to solve DC and DQA tasks, and compare the performances to well-known natural images classification models and visual question answering models. Our experiments show the effectiveness of the proposed DPN-based models on diagram understanding tasks, also indicate that our dataset is more complex compared to previous natural image understanding datasets. The presented dataset opens new challenges for research in diagram understanding, and the DPN method provides a novel perspective for studying such data. Our dataset can be available from https://github.com/WayneWong97/CSDia.
- 30 July 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 16 (6) , 1-20
Open AccessComputer Science Diagram Understanding with Topology Parsing
Diagram is a special form of visual expression for representing complex concepts, logic, and knowledge, which widely appears in educational scenes such as textbooks, blogs, and encyclopedias. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. In this article, we construct the first novel geometric type of diagrams dataset in Computer Science field, which has more abstract expressions and complex logical relations. The dataset has exhaustive annotations of objects and relations for about 1,300 diagrams and 3,500 question-answer pairs. We introduce the tasks of diagram classification (DC) and diagram question answering (DQA) based on the new dataset, and propose the Diagram Paring Net (DPN) that focuses on analyzing the topological structure and text information of diagrams. We use DPN-based models to solve DC and DQA tasks, and compare the performances to well-known natural images classification models and visual question answering models. Our experiments show the effectiveness of the proposed DPN-based models on diagram understanding tasks, also indicate that our dataset is more complex compared to previous natural image understanding datasets. The presented dataset opens new challenges for research in diagram understanding, and the DPN method provides a novel perspective for studying such data. Our dataset can be available from https://github.com/WayneWong97/CSDia.
- 29 April 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (3) , 1213-1223
Auto-FSL: Searching the Attribute Consistent Network for Few-Shot Learning
Prevailing deep methods for image recognition require massive labeled samples in each visual category for training. However, large amounts of data annotations are timeconsuming, and some uncommon categories only have rare samples available. For this issue, we focus on more challenging few-shot learning (FSL) task, where just few labeled images are used in the training stage. Existing FSL models are constructed with various convolutional neural networks (CNNs), which are trained on an auxiliary base dataset and evaluated for new fewshot predictions on a novel dataset. The performance of these models is difficult to break through because of the domain shift between base and novel datasets and the monotonous network architectures. Considering that, we propose a novel automatic attribute consistent network called Auto-ACNet to overcome the above problems. On one hand, Auto-ACNet utilizes the attribute information about base and novel categories to guide the procedure of representation learning. It introduces the consistent and non-consistent subnets to capture the common and different attributes of image pair, which helps to mitigate the domain shift problem. On the other hand, the architecture of Auto-ACNet is searched with the popular neural architecture search (NAS) technique DARTS, for obtaining a superior FSL network automatically. And the DARTS’s search space is improved by adding the position-aware module to extract the attribute characteristics better. Extensive experimental results on two datasets indicate that the proposed Auto-ACNet achieves significant improvement over the state-of-the-art competitors in this literature.
- 29 April 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (3) , 1213-1223
Auto-FSL: Searching the Attribute Consistent Network for Few-Shot Learning
Prevailing deep methods for image recognition require massive labeled samples in each visual category for training. However, large amounts of data annotations are timeconsuming, and some uncommon categories only have rare samples available. For this issue, we focus on more challenging few-shot learning (FSL) task, where just few labeled images are used in the training stage. Existing FSL models are constructed with various convolutional neural networks (CNNs), which are trained on an auxiliary base dataset and evaluated for new fewshot predictions on a novel dataset. The performance of these models is difficult to break through because of the domain shift between base and novel datasets and the monotonous network architectures. Considering that, we propose a novel automatic attribute consistent network called Auto-ACNet to overcome the above problems. On one hand, Auto-ACNet utilizes the attribute information about base and novel categories to guide the procedure of representation learning. It introduces the consistent and non-consistent subnets to capture the common and different attributes of image pair, which helps to mitigate the domain shift problem. On the other hand, the architecture of Auto-ACNet is searched with the popular neural architecture search (NAS) technique DARTS, for obtaining a superior FSL network automatically. And the DARTS’s search space is improved by adding the position-aware module to extract the attribute characteristics better. Extensive experimental results on two datasets indicate that the proposed Auto-ACNet achieves significant improvement over the state-of-the-art competitors in this literature.