Multilingual is not enough: BERT for Finnish
Authors:
- Jenna KanervaJKJenna Kanerva
- Rami IloRIRami Ilo
- Jouni LuomaJLJouni Luoma
Preprint
Abstract
Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at https://turkunlp.org/finbert .Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- SciBERT: A Pretrained Language Model for Scientific Text2019
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference2018
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- 82 Treebanks, 34 Models: Universal Dependency Parsing with Multi-Treebank Models2018
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Bag of Tricks for Efficient Text Classification2017
- Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature RepresentationsTransactions of the Association for Computational Linguistics, 2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Building the essential resources for Finnish: the Turku Dependency TreebankLanguage Resources and Evaluation, 2013