Computer Science Diagram Understanding with Topology Parsing
Open Access
- 30 July 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 16 (6) , 1-20
Abstract
Diagram is a special form of visual expression for representing complex concepts, logic, and knowledge, which widely appears in educational scenes such as textbooks, blogs, and encyclopedias. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. In this article, we construct the first novel geometric type of diagrams dataset in Computer Science field, which has more abstract expressions and complex logical relations. The dataset has exhaustive annotations of objects and relations for about 1,300 diagrams and 3,500 question-answer pairs. We introduce the tasks of diagram classification (DC) and diagram question answering (DQA) based on the new dataset, and propose the Diagram Paring Net (DPN) that focuses on analyzing the topological structure and text information of diagrams. We use DPN-based models to solve DC and DQA tasks, and compare the performances to well-known natural images classification models and visual question answering models. Our experiments show the effectiveness of the proposed DPN-based models on diagram understanding tasks, also indicate that our dataset is more complex compared to previous natural image understanding datasets. The presented dataset opens new challenges for research in diagram understanding, and the DPN method provides a novel perspective for studying such data. Our dataset can be available from https://github.com/WayneWong97/CSDia.Keywords
Funding Information
- National key research and development program (2020AAA0108800)
- National Natural Science Foundation of China (62137002, 61937001,62176209, 62176207, 62172324, 62192781, 62106190, and 62050194)
- Innovative Research Group of the National Natural Science Foundation of China (61721002)
- Innovation Research Team of Ministry of Education (IRT 17R86)
- Xi’an Jiaotong University, China Postdoctoral Science Foundation (2020M683493)
- China Knowledge Centre for Engineering Science and Technology, the Fundamental Research Funds for the Central Universities (xhj032021013-02)
This publication has 28 references indexed in Scilit:
- A Diagram is Worth a Dozen Images2016
- Deep Residual Learning for Image Recognition2016
- Semantic Parsing to Probabilistic Programs for Situated Question Answering2016
- VQA: Visual Question Answering2015
- Fully convolutional networks for semantic segmentation2015
- Glove: Global Vectors for Word Representation2014
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010
- A survey of image classification methods and techniques for improving classification performanceInternational Journal of Remote Sensing, 2007
- Extraction,layout analysis and classification of diagrams in PDF documents2005
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998