MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
Authors:
Preprint
Abstract
We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Datasheets for datasetsCommunications of the ACM, 2021
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- BERTimbau: Pretrained BERT Models for Brazilian Portuguese2020
- GREEK-BERT: The Greeks visiting Sesame Street2020
- How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence2020
- On the Cross-lingual Transferability of Monolingual Representations2020
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- Recurrent neural network-based models for recognizing requisite and effectuation parts in legal textsArtificial Intelligence and Law, 2018
- Deep Residual Learning for Image Recognition2016
- Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain2010