Harder Task Needs More Experts: Dynamic Routing in MoE Models
Authors:
- Zhenwei AnZAZhenwei An
- Yansong FengYFYansong Feng
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 12883-12895
Abstract
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models2024
- Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models2024
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them2023
- On the effect of dropping layers of pre-trained transformer modelsComputer Speech & Language, 2022
- PIQA: Reasoning about Physical Commonsense in Natural LanguageProceedings of the AAAI Conference on Artificial Intelligence, 2020
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge2019
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991