Extrapolating Large Language Models to Non-English by Aligning Languages
Preprint
Abstract
Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Large Language Models are not Fair Evaluators2024
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis2024
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data ParallelProceedings of the VLDB Endowment, 2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting2023
- ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022