How Do Vision-Language Models Process Conflicting Information Across Modalities?
Preprint
Abstract
AI models are increasingly required to be multimodal, integrating disparate input streams into a coherent state representation on which subsequent behaviors and actions can be based. This paper seeks to understand how such models behave when input streams present conflicting information. Focusing specifically on vision-language models, we provide inconsistent inputs (e.g., an image of a dog paired with the caption "A photo of a cat") and ask the model to report the information present in one of the specific modalities (e.g., "What does the caption say / What is in the image?"). We find that models often favor one modality over the other, e.g., reporting the image regardless of what the caption says, but that different models differ in which modality they favor. We find evidence that the behaviorally preferred modality is evident in the internal representational structure of the model, and that specific attention heads can restructure the representations to favor one modality over the other. Moreover, we find modality-agnostic "router heads" which appear to promote answers about the modality requested in the instruction, and which can be manipulated or transferred in order to improve performance across datasets and modalities. Together, the work provides essential steps towards identifying and controlling if and how models detect and resolve conflicting signals within complex multimodal environments.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Benchmarking Multi-Modal Semantic Segmentation Under Sensor Failures: Missing and Noisy Modality Robustness2025
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text?2025
- Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models2025
- Improved Baselines with Visual Instruction Tuning2024
- On the Robustness of Large Multimodal Models Against Image Adversarial Attacks2024
- On the Adversarial Robustness of Multi-Modal Foundation Models2023
- Multimodal biomedical AINature Medicine, 2022
- Probing Classifiers: Promises, Shortcomings, and AdvancesComputational Linguistics, 2022
- Does Vision-and-Language Pretraining Improve Lexical Grounding?2021
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009