Parameter-Efficient Fine-Tuning Design Spaces
Authors:
- Jiaao ChenJCJiaao Chen
- Aston ZhangAZAston Zhang
- Diyi YangDYDiyi Yang
Preprint
Abstract
Parameter-efficient fine-tuning aims to achieve performance comparable to fine-tuning, using fewer trainable parameters. Several strategies (e.g., Adapters, prefix tuning, BitFit, and LoRA) have been proposed. However, their designs are hand-crafted separately, and it remains unclear whether certain design patterns exist for parameter-efficient fine-tuning. Thus, we present a parameter-efficient fine-tuning design paradigm and discover design patterns that are applicable to different experimental settings. Instead of focusing on designing another individual tuning strategy, we introduce parameter-efficient fine-tuning design spaces that parameterize tuning structures and tuning strategies. Specifically, any design space is characterized by four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment. Starting from an initial design space, we progressively refine the space based on the model quality of each design choice and make greedy selection at each stage over these four components. We discover the following design patterns: (i) group layers in a spindle pattern; (ii) allocate the number of trainable parameters to layers uniformly; (iii) tune all the groups; (iv) assign proper tuning strategies to different groups. These design patterns result in new parameter-efficient fine-tuning methods. We show experimentally that these methods consistently and significantly outperform investigated parameter-efficient fine-tuning strategies across different backbone models and different tasks in natural language processing.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Improving task generalization via unified schema promptAI Open, 2023
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference2021
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning2021
- Parameter-Efficient Transfer Learning with Diff Pruning2021
- SpanBERT: Improving Pre-training by Representing and Predicting SpansTransactions of the Association for Computational Linguistics, 2020
- Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning2020
- Masking as an Efficient Alternative to Finetuning for Pretrained Language Models2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- On Network Design Spaces for Visual Recognition2019