BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages
Authors:
Preprint
Abstract
Large language models (LLMs) demonstrate promising translation performance among various natural languages. However, many LLMs especially the open-sourced ones, such as BLOOM and LLaMA, are English-dominant and support only dozens of natural languages, making the potential of LLMs on language translation less explored. In this work, we present BigTranslate which adapts LLaMA that covers only 20 languages and enhances it with multilingual translation capability on more than 100 languages. BigTranslate is built upon LLaMA-13B and it is optimized in three steps. First, we continue training LLaMA with massive Chinese monolingual data. Second, we continue training the model with a large-scale parallel dataset that covers 102 natural languages. Third, we instruct-tune the foundation model with multilingual translation instructions, leading to our BigTranslate model. The preliminary experiments on multilingual translation show that BigTranslate performs comparably with ChatGPT and Google Translate in many languages and even outperforms ChatGPT in 8 language pairs. We release the BigTranslate model and hope it can advance the research progress.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis2024
- Document-Level Machine Translation with Large Language Models2023
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions2022
- GLM: General Language Model Pretraining with Autoregressive Blank Infilling2022
- CLUE: A Chinese Language Understanding Evaluation Benchmark2020
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation2020
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017
- Neural Machine Translation of Rare Words with Subword Units2016