TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction
Authors:
- Kuan-Hao HuangKHKuan-Hao Huang
- I-Hung HsuIHI-Hung Hsu
- Tanmay ParekhTPTanmay Parekh
- Zhiyu XieZXZhiyu Xie
- Zixuan ZhangZZZixuan Zhang
- Premkumar NatarajanPNPremkumar Natarajan
- Kai-Wei ChangKCKai-Wei Chang
- Heng JiHJHeng Ji
Preprint
Abstract
Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 14 datasets spanning seven diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research.Keywords
All Related Versions
This publication has 66 references indexed in Scilit:
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- CLIP-Event: Connecting Text and Images with Event Structures2022
- ASER: A Large-scale Eventuality Knowledge Graph2020
- CASIE: Extracting Cybersecurity Event Information from TextProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Stanza: A Python Natural Language Processing Toolkit for Many Human Languages2020
- Cross-media Structured Common Space for Multimedia Event Extraction2020
- Event Detection via Gated Multilingual Attention MechanismProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Event extraction across multiple levels of biological organizationBioinformatics, 2012