A Reevaluation of Event Extraction: Past, Present, and Future Challenges
Authors:
- Kuan-Hao HuangKHKuan-Hao Huang
- I-Hung HsuIHI-Hung Hsu
- Tanmay ParekhTPTanmay Parekh
- Zhiyu XieZXZhiyu Xie
- Zixuan ZhangZZZixuan Zhang
- Premkumar NatarajanPNPremkumar Natarajan
- Kai-Wei ChangKCKai-Wei Chang
- Heng JiHJHeng Ji
Preprint
Abstract
Event extraction has attracted much attention in recent years due to its potential for many applications. However, recent studies observe some evaluation challenges, suggesting that reported scores might not reflect the true performance. In this work, we first identify and discuss these evaluation challenges, including the unfair comparisons resulting from different assumptions about data or different data preprocessing steps, the incompleteness of the current evaluation framework leading to potential dataset bias or data split bias, and low reproducibility of prior studies. To address these challenges, we propose TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE contains standardized data preprocessing scripts and splits for more than ten datasets across different domains. In addition, we aggregate and re-implement over ten event extraction approaches published in recent years and conduct a comprehensive reevaluation. Finally, we explore the capability of large language models in event extraction and discuss some future challenges. We expect TextEE will serve as a reliable benchmark for event extraction, facilitating future research in the field.Keywords
All Related Versions
This publication has 61 references indexed in Scilit:
- CASIE: Extracting Cybersecurity Event Information from TextProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection2020
- Event Extraction as Machine Reading Comprehension2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Multi-Sentence Argument Linking2020
- Stanza: A Python Natural Language Processing Toolkit for Many Human Languages2020
- A Joint Neural Model for Information Extraction with Global Features2020
- Cross-media Structured Common Space for Multimedia Event Extraction2020
- Event Detection via Gated Multilingual Attention MechanismProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Freebase2008