When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

Authors:
Top Cited Papers
Open Access
Publisher website
Google Scholar
Abstract
The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.
Funding Information
  • SIRG - CityU Strategic Interdisciplinary Research Grant (No.7020046, No.7020074)
  • CCF-Tencent Open Fund
  • CityU - HKIDS Early Career Research Grant (No.9360163)
  • APRC - CityU New Research Initiatives (No.9610565, Start-up Grant for New Faculty of CityU)
  • Hong Kong Environmental and Conservation Fund (No. 88/2022)
  • Research Impact Fund (No.R1015-23)
  • Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No.ITS/034/22MS)

This publication has 28 references indexed in Scilit: