WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Preprint
Abstract
We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages. We do not limit the the extraction process to alignments with English, but systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus of parallel sentences is freely available at https://github.com/facebookresearch/LASER/tree/master/tasks/WikiMatrix. To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings2019
- Filtering and Mining Parallel Data in a Joint Multilingual Space2018
- Bag of Tricks for Efficient Text Classification2017
- Set-Theoretic Alignment for Comparable Corpora2016
- Product Quantization for Nearest Neighbor SearchIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- The Web as a Parallel CorpusComputational Linguistics, 2003
- The United Nations Parallel Corpus v1.0