Scene Graph Enhanced Multi-Perspective Gated Interaction Framework for Multimodal Sarcasm Detection

Authors:
  • ZZZihao Zheng
    Zihao Zheng
  • XLXiangyuan Lan
    Xiangyuan Lan
  • YLYunfei Long
    Yunfei Long
  • XZXuhan Zhu
    Xuhan Zhu
  • DJDongmei Jiang
    Dongmei Jiang
  • MLMing Liu
    Ming Liu
  • RXRuifeng Xu
    Ruifeng Xu

Abstract
Sarcasm, a complex linguistic phenomenon where literal and intended meanings differ, is widespread on social media. Effective sarcasm detection is vital for interpreting user intent and behavior accurately. Multimodal sarcasm detection aims to identify sarcastic expressions from given multimodal data, primarily images and text. Although various effective methods have been proposed, especially graph-based approaches, they often fail to thoroughly explore intra-modal inconsistencies, particularly within the image modality, thereby limiting detection performance. To address this limitation, we introduce scene graph knowledge, a directed graph that clearly depicts intra-modal semantic relationships. Based on this, we propose a Scene Graph Enhanced Multi-Perspective Gated Interaction Framework for multimodal sarcasm detection. This method first constructs visual and textual relationship graphs, incorporating semantic relationships as nodes to enhance the model's focus on intra-modal relationship modeling. The two graphs are subsequently fused, and cross-modal interaction nodes are added to form a cross-modal interaction graph. We then employ a multi-perspective gated interaction mechanism, centering on text, image, visual relationship graph, and textual relationship graph, to aggregate useful information from adjacent modalities. This comprehensive approach provides a deep and nuanced understanding of sarcastic expressions. Experimental results demonstrate that our proposed method effectively captures inconsistencies within and between modalities, achieving superior performance compared to existing state-of-the-art methods.

This publication has 19 references indexed in Scilit: