ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
Preprint
Abstract
Recently, large language models have shown remarkable reasoning capabilities through long-chain reasoning before responding. However, how to extend this capability to visual reasoning tasks remains an open challenge. Existing multimodal reasoning approaches transfer such visual reasoning task into textual reasoning task via several image-to-text conversions, which often lose critical structural and semantic information embedded in visualizations, especially for tasks like chart question answering that require a large amount of visual details. To bridge this gap, we propose ChartReasoner, a code-driven novel two-stage framework designed to enable precise, interpretable reasoning over charts. We first train a high-fidelity model to convert diverse chart images into structured ECharts codes, preserving both layout and data semantics as lossless as possible. Then, we design a general chart reasoning data synthesis pipeline, which leverages this pretrained transport model to automatically and scalably generate chart reasoning trajectories and utilizes a code validator to filter out low-quality samples. Finally, we train the final multimodal model using a combination of supervised fine-tuning and reinforcement learning on our synthesized chart reasoning dataset and experimental results on four public benchmarks clearly demonstrate the effectiveness of our proposed ChartReasoner. It can preserve the original details of the charts as much as possible and perform comparably with state-of-the-art open-source models while using fewer parameters, approaching the performance of proprietary systems like GPT-4o in out-of-domain settings.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- A Multimodal LLM for Chart Understanding and Generation2025
- EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart UnderstandingProceedings of the AAAI Conference on Artificial Intelligence, 2025
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- Improved Baselines with Visual Instruction Tuning2024
- ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules2023
- RealCQA: Scientific Chart Question Answering as a Test-Bed for First-Order Logic2023
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning2022
- PlotQA: Reasoning over Scientific Plots2020
- LEAF-QA: Locate, Encode & Attend for Figure Question Answering2020
- DVQA: Understanding Data Visualizations via Question Answering2018