LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages
Preprint
Abstract
Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages. Through a comprehensive analysis of training strategies, such as vocabulary expansion and data augmentation, we develop LLaMAX. Remarkably, without sacrificing its generalization ability, LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs~(by more than 10 spBLEU points) and performs on-par with specialized translation model~(M2M-100-12B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model. The code~\footnote{\url{https://github.com/CONE-MT/LLaMAX/.}} and models~\footnote{\url{https://huggingface.co/LLaMAX/.}} are publicly available.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- The UMD Submission to the Explainable MT Quality Estimation Shared Task: Combining Explanation Models with Sequence Labeling2021
- It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning2021
- Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning2021
- Contrastive Learning for Many-to-many Multilingual Neural Machine Translation2021
- Neural Machine Translation with Byte-Level SubwordsProceedings of the AAAI Conference on Artificial Intelligence, 2020
- TICO-19: the Translation Initiative for COvid-192020
- COMET: A Neural Framework for MT Evaluation2020
- XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019