Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models
Preprint
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual understanding tasks, yet they often suffer from object hallucinations--generating descriptions of objects that are inconsistent with or entirely absent from the input. This issue is closely related to dataset biases, where frequent co-occurrences of objects lead to entangled semantic representations across modalities. As a result, models may erroneously activate object representations that are commonly associated with the input but not actually present. To address this, we propose a causality-driven disentanglement framework that mitigates hallucinations through causal intervention. Our approach includes a Causal-Driven Projector in the visual pathway and a Causal Intervention Module integrated into the final transformer layer of the language model. These components work together to reduce spurious correlations caused by biased training data. Experimental results show that our method significantly reduces hallucinations while maintaining strong performance on multiple multimodal benchmarks. Visualization analyses further confirm improved separability of object representations. The code is available at: https://github.com/IgniSavium/Causal-LLaVAKeywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Woodpecker: hallucination correction for multimodal large language modelsScience China Information Sciences, 2024
- Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding2024
- Aligning Large Multimodal Models with Factually Augmented RLHF2024
- Unbiased Scene Graph Generation via Two-Stage Causal ModelingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Deconfounded Image Captioning: A Causal RetrospectIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- A Survey on Causal InferenceACM Transactions on Knowledge Discovery From Data, 2021
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- The Deep Regression Bayesian Network and Its Applications: Probabilistic Deep Learning for Computer VisionIEEE Signal Processing Magazine, 2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Causal feature learning: an overviewBehaviormetrika, 2016