Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization
Preprint
Abstract
Multilingual sentence representations are the foundation for similarity-based bitext mining, which is crucial for scaling multilingual neural machine translation (NMT) system to more languages. In this paper, we introduce MuSR: a one-for-all Multilingual Sentence Representation model that supports more than 220 languages. Leveraging billions of English-centric parallel corpora, we train a multilingual Transformer encoder, coupled with an auxiliary Transformer decoder, by adopting a multilingual NMT framework with CrossConST, a cross-lingual consistency regularization technique proposed in Gao et al. (2023). Experimental results on multilingual similarity search and bitext mining tasks show the effectiveness of our approach. Specifically, MuSR achieves superior performance over LASER3 (Heffernan et al., 2022) which consists of 148 independent multilingual sentence encoders.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization2023
- Language-agnostic BERT Sentence Embedding2022
- WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia2021
- CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web2021
- CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs2020
- Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation2020
- Multilingual Universal Sentence Encoder for Semantic Retrieval2020
- ParaCrawl: Web-Scale Acquisition of Parallel Corpora2020
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018