LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Preprint
Abstract
Fine-tuning large pre-trained models on downstream tasks has been adopted in a variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficient transfer learning (PETL) techniques allow updating a small subset of parameters (e.g. only using 2% of parameters) inside a pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires backpropagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, we train a ladder side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (T5 and CLIP-T5) on both NLP (GLUE) and vision-and-language (VQA, GQA, NLVR2 , MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models, attaining better GLUE performance than full fine-tuning and other PETL methods. The accuracy-efficiency trade-off also holds on VL tasks.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- $\cal{Y}$-Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learningFrontiers of Computer Science, 2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Visual Prompt Tuning2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Masked Autoencoders Are Scalable Vision Learners2022
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Parameter-Efficient Transfer Learning with Diff Pruning2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Neural Network Acceptability JudgmentsTransactions of the Association for Computational Linguistics, 2019