XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 11664-11674
Abstract
Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, Zenglin Xu. Findings of the Association for Computational Linguistics ACL 2024. 2024.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models2024
- Adaptive Gating in Mixture-of-Experts based Language Models2023
- Knowledge Neurons in Pretrained Transformers2022
- MoEfication: Transformer Feed-forward Layers are Mixtures of Experts2022
- Transformer Feed-Forward Layers Are Key-Value Memories2021
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019