Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
Authors:
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 784-801
Abstract
Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, Yu Wu. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- LoRA dropout as a sparsity regularizer for overfitting reductionKnowledge-Based Systems, 2025
- StableMoE: Stable Routing Strategy for Mixture of Experts2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning2021
- Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation2021
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning2021
- Parameter-Efficient Transfer Learning with Diff Pruning2021
- CLUE: A Chinese Language Understanding Evaluation Benchmark2020
- ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization2020