Task Relation Distillation and Prototypical Pseudo Label for Incremental Named Entity Recognition
Open Access
- 21 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3319-3329
Abstract
Incremental Named Entity Recognition (INER) involves the sequential learning of new entity types without accessing the training data of previously learned types. However, INER faces the challenge of catastrophic forgetting specific for incremental learning, further aggravated by background shift (i.e., old and future entity types are labeled as the non-entity type in the current task). To address these challenges, we propose a method called task Relation Distillation and Prototypical pseudo label (RDP) for INER. Specifically, to tackle catastrophic forgetting, we introduce a task relation distillation scheme that serves two purposes: 1) ensuring inter-task semantic consistency across different incremental learning tasks by minimizing inter-task relation distillation loss, and 2) enhancing the model's prediction confidence by minimizing intra-task self-entropy loss. Simultaneously, to mitigate background shift, we develop a prototypical pseudo label strategy that distinguishes old entity types from the current non-entity type using the old model. This strategy generates high-quality pseudo labels by measuring the distances between token embeddings and type-wise prototypes. We conducted extensive experiments on ten INER settings of three benchmark datasets (i.e., CoNLL2003, I2B2, and OntoNotes5). The results demonstrate that our method achieves significant improvements over the previous state-of-the-art methods, with an average increase of 6.08% in Micro F1 score and 7.71% in Macro F1 score.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training2018
- Memory Aware Synapses: Learning What (not) to Forget2018
- Lifelong Machine Learning, Second EditionSynthesis Lectures on Artificial Intelligence and Machine Learning, 2018
- Learning without ForgettingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF2016
- Neural Architectures for Named Entity Recognition2016
- Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)Journal of the American Medical Informatics Association, 2010
- Named entity recognition in query2009
- Catastrophic Forgetting, Rehearsal and PseudorehearsalConnection Science, 1995