Caption-Aware Multimodal Relation Extraction with Mutual Information Maximization
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1148-1157
Abstract
Multimodal Relation Extraction (MRE) has achieved great improvements. However, modern MRE models are easily affected by irrelevant objects during multimodal alignment which are called error sensitivity issues. The main reason is that visual features are not fully aligned with textual features and the reasoning process may suppress redundant and noisy information at the risk of losing critical information. In light of this, we propose a Caption-Aware Multimodal Relation Extraction Network with Mutual Information Maximization (CAMIM). Specifically, we first generate detailed image captions through the Large Language Model (LLM). Then, the Caption-Aware Module (CAM) hierarchically aligns the fine-grained visual entities and textual entities for reasoning. In addition, for preserving crucial information within different modalities, we leverage a Mutual Information Maximization method to regulate the multimodal reasoning module. Experiments show that our model outperforms the state-of-the-art MRE models on the benchmark dataset MNRE. Further ablation studies prove the pluggable and effective performance of our Caption-Aware Module and Mutual Information Maximization method. Our code is available at https://github.com/zefanZhang-cn/CAMIM.Keywords
Funding Information
- National Natural Science Foundation of China (U21A20390)
- Development Project of Jilin Province of China (20240601039RC)
- Fundamental Research Funds for the Central University, JLU
This publication has 32 references indexed in Scilit:
- Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering2024
- Building Lane-Level Maps from Aerial Images2024
- Enhancing Multimodal Entity and Relation Extraction With Variational Information BottleneckIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space2023
- TSVFN: Two-Stage Visual Fusion Network for multimodal relation extractionInformation Processing & Management, 2023
- MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- A Fast and Accurate One-Stage Approach to Visual Grounding2019
- Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition2019
- Deep Residual Learning for Image Recognition2016