SDLE: Shallow-deep learning for out-of-context misinformation detection using multimodal large language models

Authors:
  • JWJunjie Wu
    Junjie Wu
  • YFYumeng Fu
    Yumeng Fu
  • CFChuanliu Fan
    Chuanliu Fan
  • GFGuohong Fu
    Guohong Fu

Abstract
Out-Of-Context (OOC) misinformation detection aims to identify whether a genuine image is repurposed in a misleading textual content. Such misinformation arises when genuine images are detached from their original contexts and paired with semantically related but false texts, making detection based solely on image-text pairs challenging. Existing methods primarily exploit internal knowledge from image-text pairs or external knowledge from retrieved evidence, yet they often fail to establish deep alignment between these sources due to limited interaction. To address these limitations, we propose SDLE, a novel Shallow-Deep Learning framework that enhances Multimodal Large Language Models (MLLMs) for OOC misinformation detection. Specifically, SDLE first retrieves external evidence from various sources to gather contextual information related to image-text pairs. To build a shallow relationship between internal and external knowledge, SDLE designs a prompt learning method to reduce noise and capture key evidence. Subsequently, to build a deep relationship between them, SDLE aligns MLLMs for evidence-based rationale generation. Furthermore, we adopt instruction tuning techniques to empower the model with both judgments and explanations in the realm of OOC misinformation detection. Comprehensive experiments on two widely used benchmark datasets, NewsCLIPpings and VERITE, show that SDLE consistently outperforms existing methods, achieving F1-score improvements of 1.4% and 2.0%, respectively.