UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Authors:
- Ahmed MasryAMAhmed Masry
- Parsa KavehzadehPKParsa Kavehzadeh
- Enamul HoqueEHEnamul Hoque
- Shafiq JotySJShafiq Joty
Preprint
Abstract
Charts are very popular for analyzing data, visualizing key insights and answering complex reasoning questions about data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently such as chart question answering and chart summarization. However, most of the methods that solve these tasks use pretraining on language or vision-language tasks that do not attempt to explicitly model the structure of the charts (e.g., how data is visually encoded and how chart elements are related to each other). To address this, we first build a large corpus of charts covering a wide variety of topics and visual styles. We then present UniChart, a pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual elements of charts and then uses a chart-grounded text decoder to generate the expected output in natural language. We propose several chart-specific pretraining tasks that include: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart understanding and reasoning skills. We find that pretraining the model on a large corpus with chart-specific low- and high-level tasks followed by finetuning on three down-streaming tasks results in state-of-the-art performance on three downstream tasks.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Unifying Vision, Text, and Layout for Universal Document Processing2023
- Scene Text Recognition with Permuted Autoregressive Sequence Models2022
- Towards Natural Language-Based Visualization AuthoringIEEE Transactions on Visualization and Computer Graphics, 2022
- Striking a Balance: Reader Takeaways and Preferences when Integrating Text and ChartsIEEE Transactions on Visualization and Computer Graphics, 2022
- Chart Question Answering: State of the Art and Future DirectionsComputer Graphics Forum, 2022
- Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic ContentIEEE Transactions on Visualization and Computer Graphics, 2021
- Calliope: Automatic Visual Data Story Generation from a SpreadsheetIEEE Transactions on Visualization and Computer Graphics, 2020
- DVQA: Understanding Data Visualizations via Question Answering2018
- Vega-Lite: A Grammar of Interactive GraphicsIEEE Transactions on Visualization and Computer Graphics, 2016
- Show Me: Automatic Presentation for Visual AnalysisIEEE Transactions on Visualization and Computer Graphics, 2007