Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
Open Access
- 1 January 2025
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 17948-17963
Abstract
Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.Keywords
This publication has 18 references indexed in Scilit:
- A survey on large language models with multilingualism: recent advances and new frontiersArtificial Intelligence Review, 2026
- MoE-LPR: Multilingual Extension of Large Language Models Through Mixture-of-Experts with Language Priors RoutingProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge NeuronsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models2024
- LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin2024
- ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning2023
- Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback2023
- Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models2023
- How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models2021
- COMET: A Neural Framework for MT Evaluation2020