Out-Of-Context (OOC) misinformation detection aims to identify whether a genuine image is repurposed in a misleading textual content. Such misinformation arises when genuine images are detached from their original contexts and paired with semantically related but false texts, making detection based solely on image-text pairs challenging. Existing methods primarily exploit internal knowledge from image-text pairs or external knowledge from retrieved evidence, yet they often fail to establish deep alignment between these sources due to limited interaction. To address these limitations, we propose SDLE, a novel Shallow-Deep Learning framework that enhances Multimodal Large Language Models (MLLMs) for OOC misinformation detection. Specifically, SDLE first retrieves external evidence from various sources to gather contextual information related to image-text pairs. To build a shallow relationship between internal and external knowledge, SDLE designs a prompt learning method to reduce noise and capture key evidence. Subsequently, to build a deep relationship between them, SDLE aligns MLLMs for evidence-based rationale generation. Furthermore, we adopt instruction tuning techniques to empower the model with both judgments and explanations in the realm of OOC misinformation detection. Comprehensive experiments on two widely used benchmark datasets, NewsCLIPpings and VERITE, show that SDLE consistently outperforms existing methods, achieving F1-score improvements of 1.4% and 2.0%, respectively.