Zero-Shot Cross-Lingual Named Entity Recognition via Progressive Multi-Teacher Distillation
- 23 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 4617-4630
Abstract
Cross-lingual learning aims to transfer knowledge from one natural language to another. Zero-shot cross-lingual named entity recognition (NER) tasks are to train an NER model on source languages and to identify named entities in other languages. Existing knowledge distillation-based models in a teacher-student manner leverage the unlabeled samples from the target languages and show their superiority in this setting. However, the valuable similarity information between tokens in the target language is ignored. And the teacher model trained solely on the source language generates low-quality pseudo-labels. These two facts impact the performance of cross-lingual NER. To improve the reliability of the teacher model, in this study, we first introduce one extra simple binary classification teacher model by similarity learning to measure if the inputs are from the same class. We note that this binary classification auxiliary task is easier, and the two teachers simultaneously supervise the student model for better performance. Furthermore, given such a stronger student model, we propose a progressive knowledge distillation framework that extensively fine-tunes the teacher model on the target-language pseudo-labels generated by the student model. Empirical studies on three datasets across seven different languages show that our presented model outperforms state-of-the-art methodsKeywords
Funding Information
- National Science and Technology Major (2022ZD0120202)
- National Natural Science Foundation of China (U23B2056)
- Fundamental Research Funds for the Central Universities
- State Key Laboratory of Complex & Critical Software Environment
This publication has 34 references indexed in Scilit:
- Cross-Lingual Named Entity Recognition for Heterogenous LanguagesIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- Online Knowledge Distillation via Collaborative Learning2020
- Named Entity Location Prediction Combining Twitter and WebIEEE Transactions on Knowledge and Data Engineering, 2020
- A General Framework for First Story Detection Utilizing Entities and Their RelationsIEEE Transactions on Knowledge and Data Engineering, 2020
- Person Reidentification via Structural Deep Metric LearningIEEE Transactions on Neural Networks and Learning Systems, 2018
- Deep Mutual Learning2018
- Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition2017
- Entity Synonyms for Structured Web SearchIEEE Transactions on Knowledge and Data Engineering, 2011
- A unified architecture for natural language processing2008
- SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORKInternational Journal of Pattern Recognition and Artificial Intelligence, 1993