Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles
Authors:
Preprint
Abstract
Event coreference resolution (ECR) is the task of determining whether distinct mentions of events within a multi-document corpus are actually linked to the same underlying occurrence. Images of the events can help facilitate resolution when language is ambiguous. Here, we propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. As existing ECR benchmark datasets rarely provide images for all event mentions, we augment the popular ECB+ dataset with event-centric images scraped from the internet and generated using image diffusion models. We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty. We evaluate on 2 datasets: the augmented ECB+, and AIDA Phase 1. Our ensemble systems using cross-modal linear mapping establish an upper limit (91.9 CoNLL F1) on ECB+ ECR performance given the preprocessing assumptions used, and establish a novel baseline on AIDA Phase 1. Our results demonstrate the utility of multimodal information in ECR for certain challenging coreference problems, and highlight a need for more multimodal resources in the coreference resolution space.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese2023
- 2*n is better than is better than n2: Decomposing Event Coreference Resolution into Two Tractable Problems: Decomposing Event Coreference Resolution into Two Tractable Problems2023
- Canonical Face EmbeddingsIEEE Transactions on Biometrics, Behavior, and Identity Science, 2022
- Generalizing Cross-Document Event Coreference Resolution Across Multiple CorporaComputational Linguistics, 2021
- CDLM: Cross-Document Language Modeling2021
- Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference2021
- Cross-document Coreference Resolution over Predicted Mentions2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection2019
- Translating Granularity of Event Slots into Features for Event Coreference Resolution.2015