LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training
Top Cited Papers
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 15913-15923
Abstract
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, Yu Cheng. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging2024
- Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models2024
- Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?2024
- MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation2022
- MoEfication: Transformer Feed-forward Layers are Mixtures of Experts2022
- LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Crowdsourcing Multiple Choice Science Questions2017