Alignment Relation is What You Need for Diagram Parsing
- 13 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 2131-2144
Abstract
As a knowledge carrier, the diagram is widely distributed in many aspects of human life, such as textbooks, architectural drawings, and documents. Different from natural images, representations of visual elements in the diagram are sparser, and similar visual representations can reflect dissimilar semantics. Thus, current methods fail to capture the visual elements with precise semantics. To address this issue, regarding the aligned visual and textual elements as pairs is the way to assign the precise semantics of textual elements to visual elements. We build the first diagram dataset named align diagram element (ADE), which includes annotations for alignment relations between visual and textual elements. And we propose a visual-textual alignment model (VTAM) including graph construction and optimal aligning phases. In the graph construction phase, the relational graphs are constructed between different elements with four relational operators. The relational operators are designed to measure the relations between different elements, according to distance, connection line, inclusion, and feature similarity. In the optimal aligning phase, the representation at each visual-textual pair is improved as a weighted sum of the representations on all relational graphs. Experimental results show that our VTAM achieves a significant improvement of 10.9% on mean test folds of the ADE dataset than the current best competitor. In order to explore the role of alignment relations in diagram parsing, we introduce VTAM to diagram-related tasks, such as diagram question answering (DQA). And we achieve 2.8% to 5.9% and 4.6% to 5.1% improvements on AI2D and Foodwebs after adding VTAM. Our dataset and code are released at: https://github.com/ADE-dataset/ADE-dataset.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFC3303600)
- National Natural Science Foundation of China (62137002, 62293553, 62293554, 62192781, 62176209, 62106190, 61937001)
- “LENOVO-XJTU” Intelligent Industry Joint Laboratory Project, Natural Science Basic Research Program of Shaanxi (2023-JC-YB-593)
- Youth Innovation Team of Shaanxi Universities, XJTU Teaching Reform Research Project “Acquisition Learning Based on Knowledge Forest”
- Shaanxi Undergraduate and Higher Education Teaching Reform Research Program (23BY195)
This publication has 38 references indexed in Scilit:
- My Teacher Is a Machine: Understanding Students’ Perceptions of AI Teaching Assistants in Online EducationInternational Journal of Human–Computer Interaction, 2020
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Textbook Question Answering Under Instructor Guidance with Memory Networks2018
- Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams2018
- Non-local Neural Networks2018
- Dual Attention Networks for Multimodal Reasoning and Matching2017
- Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension2017
- A Diagram is Worth a Dozen Images2016
- Deep Residual Learning for Image Recognition2016
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955