Scene Graph Enhanced Multi-Perspective Gated Interaction Framework for Multimodal Sarcasm Detection
Authors:
- Zhiyuan WenZWZhiyuan Wen
- Zihao ZhengZZZihao Zheng
- Xiangyuan LanXLXiangyuan Lan
- Yunfei LongYLYunfei Long
- Xuhan ZhuXZXuhan Zhu
- Dongmei JiangDJDongmei Jiang
- Ming LiuMLMing Liu
- Ruifeng XuRXRuifeng Xu
Preprint
- 27 June 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Sarcasm, a complex linguistic phenomenon where literal and intended meanings differ, is widespread on social media. Effective sarcasm detection is vital for interpreting user intent and behavior accurately. Multimodal sarcasm detection aims to identify sarcastic expressions from given multimodal data, primarily images and text. Although various effective methods have been proposed, especially graph-based approaches, they often fail to thoroughly explore intra-modal inconsistencies, particularly within the image modality, thereby limiting detection performance. To address this limitation, we introduce scene graph knowledge, a directed graph that clearly depicts intra-modal semantic relationships. Based on this, we propose a Scene Graph Enhanced Multi-Perspective Gated Interaction Framework for multimodal sarcasm detection. This method first constructs visual and textual relationship graphs, incorporating semantic relationships as nodes to enhance the model's focus on intra-modal relationship modeling. The two graphs are subsequently fused, and cross-modal interaction nodes are added to form a cross-modal interaction graph. We then employ a multi-perspective gated interaction mechanism, centering on text, image, visual relationship graph, and textual relationship graph, to aggregate useful information from adjacent modalities. This comprehensive approach provides a deep and nuanced understanding of sarcastic expressions. Experimental results demonstrate that our proposed method effectively captures inconsistencies within and between modalities, achieving superior performance compared to existing state-of-the-art methods.Keywords
This publication has 19 references indexed in Scilit:
- A survey of automatic sarcasm detection: Fundamental theories, formulation, datasets, detection methods, and opportunitiesNeurocomputing, 2024
- Multi-modal sarcasm detection based on Multi-Channel Enhanced Fusion modelNeurocomputing, 2024
- A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learningInformation Processing & Management, 2024
- Explainable depression detection with multi-aspect features using a hybrid deep learning model on social mediaWorld Wide Web, 2022
- Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval2020
- Sarcasm Detection with Self-matching Networks and Low-rank Bilinear Pooling2019
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Automatic Sarcasm DetectionACM Computing Surveys, 2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005