ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understanding
Authors:
Preprint
Abstract
Large Vision-Language Models (LVLMs) have recently demonstrated remarkable progress, yet hallucination remains a critical barrier, particularly in chart understanding, which requires sophisticated perceptual and cognitive abilities as well as rigorous factual accuracy. While prior work has investigated hallucinations and chart comprehension independently, their intersection remains largely unexplored. To address this gap, we present ChartHal, a benchmark that features a fine-grained taxonomy of hallucination scenarios in chart understanding, along with a human-validated dataset of 1,062 samples. Our evaluation shows that state-of-the-art LVLMs suffer from severe hallucinations on ChartHal, including proprietary models such as GPT-5 and o4-mini, which achieve only 34.46% and 22.79% accuracy, respectively. Further analysis reveals that questions involving information absent from or contradictory to charts are especially likely to trigger hallucinations, underscoring the urgent need for more robust mitigation strategies. Code and data are available at https://github.com/ymcui/ChartHal .Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Aligning Large Multimodal Models with Factually Augmented RLHF2024
- MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning2024
- Visual Instruction Tuning2023
- Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?2023
- Evaluating Object Hallucination in Large Vision-Language Models2023
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning2022
- Object Hallucination in Image Captioning2018