SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models
Preprint
Abstract
Full fine-tuning is a popular approach to adapt Transformer-based pre-trained large language models to a specific downstream task. However, the substantial requirements for computational power and storage have discouraged its widespread use. Moreover, increasing evidence of catastrophic forgetting and overparameterization in the Transformer architecture has motivated researchers to seek more efficient fine-tuning (PEFT) methods. Commonly known parameter-efficient fine-tuning methods like LoRA and BitFit are typically applied across all layers of the model. We propose a PEFT method, called Stratified Progressive Adaptation Fine-tuning (SPAFIT), based on the localization of different types of linguistic knowledge to specific layers of the model. Our experiments, conducted on nine tasks from the GLUE benchmark, show that our proposed SPAFIT method outperforms other PEFT methods while fine-tuning only a fraction of the parameters adjusted by other methods.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer2020
- Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning2020
- Neural Network Acceptability JudgmentsTransactions of the Association for Computational Linguistics, 2019
- BERT Rediscovers the Classical NLP Pipeline2019
- Dissecting Contextual Word Embeddings: Architecture and Representation2018
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- A large annotated corpus for learning natural language inference2015
- Visualizing and Understanding Convolutional Networks2014