How Do LLMs See Charts? A Comparative Study on High‐Level Visualization Comprehension in Humans and LLMs
Open Access
- 11 June 2026
- journal article
- research article
- Published by Wiley in Computer Graphics Forum
- p. e70450
Abstract
Designers often create visualizations to achieve specific high‐level analytical or communication goals. These goals require people to extract complex and interconnected data patterns. Prior perceptual studies of visualization effectiveness have focused on low‐level tasks, such as estimating statistical quantities, and have recently explored high‐level comprehension of visualization. Despite the growing use of Large Language Models (LLMs) as visualization interpreters, how their interpretations relate to human understanding or what reasoning processes underlie their responses remains insufficiently understood. In this work, we explore LLMs' comprehension of visualization, examining the alignment between designers' communicative goals and what their audience sees. We have conducted a qualitative study to investigate the gap between human interpretative strategies and the reasoning pathways of LLMs across three types of visualizations, line graphs, bar graphs, and scatterplots, to identify the high‐level patterns generated by LLMs using three prompt conditions. Our analysis results indicate that LLMs exhibit a consistent interpretative strategy that remains unchanged across prompt constraints. Furthermore, we observe two distinct approaches: humans naturally synthesize data into trend‐centric narratives, whereas LLMs persist with a structural enumeration of comparisons and numerical ranges. Lastly, we see LLMs achieve visualization comprehension through mechanisms distinct from human intuition, pointing to critical challenges and new opportunities for visualization design.Keywords
All Related Versions
Funding Information
- Pohang University of Science and Technology
- National Research Foundation of Korea (RS‐2024‐00456247)
- National Research Foundation of Korea (RS‐2023‐00218913)
- Ministry of Science and ICT, South Korea
This publication has 33 references indexed in Scilit:
- Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language ModelsIEEE Access, 2023
- Designing With Pictographs: Envision Topics Without Sacrificing UnderstandingIEEE Transactions on Visualization and Computer Graphics, 2021
- A rapidly deployed, interactive, online visualization system to support fatality management during the coronavirus disease 2019 (COVID-19) pandemicJournal of the American Medical Informatics Association, 2020
- How to evaluate data visualizations across different levels of understanding2020
- Uncertainty-Aware Visualization for Analyzing Heterogeneous Wildfire DetectionsIEEE Computer Graphics and Applications, 2019
- Evaluating the impact of visualization of wildfire hazard upon decision-making under uncertaintyInternational Journal of Geographical Information Science, 2016
- Computerized measures of visual complexityActa Psychologica, 2015
- Low-level components of analytic activity in information visualization2005
- Visual Attention: Bottom-Up Versus Top-DownCurrent Biology, 2004
- Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical MethodsJournal of the American Statistical Association, 1984