StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding
Top Cited Papers
- 1 July 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (7) , 8044-8059
Abstract
Charts are common in literature across various scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception that extracts information from the visual charts, or chart reasoning given the extracted data, e.g. in a tabular form. In this paper, we introduce StructChart, a novel framework that leverages Structured Triplet Representations (STR) to achieve a unified and label-efficient approach to chart perception and reasoning tasks, which is generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart data from the tubular form (linearized CSV) to STR, which can friendlily reduce the task gap between chart perception and reasoning. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the chart perception task performance. To augment the training, we further explore the potential of Large Language Models (LLMs) to enhance the diversity in both chart visual style and statistical information. Extensive experiments on various chart-related tasks demonstrate the effectiveness and potential of a unified chart perception-reasoning paradigm to push the frontier of chart understanding.Keywords
All Related Versions
Funding Information
- Scientific Research Innovation Capability Support Project for Young Faculty
- Ministry of Education of the People's Republic of China (SRICSPYF-ZY2025019)
- Science and Technology Commission of Shanghai Municipality (22DZ1100102)
- Shanghai Artificial Intelligence Laboratory
- Shanghai Municipal Government
- Shanghai Rising Star Program (23QD1401000)
This publication has 26 references indexed in Scilit:
- A Multimodal LLM for Chart Understanding and Generation2025
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- ShareGPT4V: Improving Large Multi-modal Models with Better Captions2024
- Improved Baselines with Visual Instruction Tuning2024
- Uni3D: A Unified Baseline for Multi-Dataset 3D Object Detection2023
- FLAVA: A Foundational Language And Vision Alignment Model2022
- Masked Autoencoders Are Scalable Vision Learners2022
- ChartReader: Automatic Parsing of Bar-Plots2021
- Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning2021
- CornerNet: Detecting Objects as Paired KeypointsInternational Journal of Computer Vision, 2019