Bin Duan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Bin Duan has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Binghao Tang
Meituan, Beijing, China
1 shared publication - Gang Zhao
Beijing University of Posts and Telecommunications, Beijing, China
1 shared publication - Huan Ouyang
Beijing University of Posts and Telecommunications, Beijing, China
1 shared publication - Lei Luo
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
1 shared publication - Si Li
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
1 shared publication - Xiaocheng Gong
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
1 shared publication
Latest Publications
- Open Access
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4183-4187
CAG: A Consistency-Adaptive Text-Image Alignment Generation for Joint Multimodal Entity-Relation Extraction
Joint Multimodal Entity-Relation Extraction (JMERE) aims to extract entity-relationship triples in texts from given image-text pairs. As a joint multimodal information extraction task, it has attracted increasing research interest. Previous works of JMERE typically utilize graph networks to align textual entities and visual objects and achieve promising performance. However, these methods do not pay attention to the inconsistency between text and image and the straight alignment could limit the performance of JMERE models. In this paper, we propose a Consistency-adaptive text-image Alignment Generation (CAG) framework for various text-image consistency scenarios. Specifically, we propose a Consistency Factor (CF) to measure the consistency between images and texts. We also design consistency-adaptive contrastive learning based on CF, which can reduce the impact of inconsistent visual and textual information. Additionally, we adopt JMERE-specifical instruction tuning for better entity-relationship triplet generation. Experimental results on the JMERE dataset demonstrate that our proposed CAG is effective and achieves state-of-the-art performance.
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4183-4187
Open AccessCAG: A Consistency-Adaptive Text-Image Alignment Generation for Joint Multimodal Entity-Relation Extraction
Joint Multimodal Entity-Relation Extraction (JMERE) aims to extract entity-relationship triples in texts from given image-text pairs. As a joint multimodal information extraction task, it has attracted increasing research interest. Previous works of JMERE typically utilize graph networks to align textual entities and visual objects and achieve promising performance. However, these methods do not pay attention to the inconsistency between text and image and the straight alignment could limit the performance of JMERE models. In this paper, we propose a Consistency-adaptive text-image Alignment Generation (CAG) framework for various text-image consistency scenarios. Specifically, we propose a Consistency Factor (CF) to measure the consistency between images and texts. We also design consistency-adaptive contrastive learning based on CF, which can reduce the impact of inconsistent visual and textual information. Additionally, we adopt JMERE-specifical instruction tuning for better entity-relationship triplet generation. Experimental results on the JMERE dataset demonstrate that our proposed CAG is effective and achieves state-of-the-art performance.