Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and Low-Rank Adaptation with only 0.94% of trainable parameters.Keywords
All Related Versions
Funding Information
- Ministry of Education
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
This publication has 14 references indexed in Scilit:
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- SUPERB: Speech Processing Universal PERformance Benchmark2021
- Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining2021
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- Multilingual Speech Recognition with Self-Attention Structured Parameterization2020
- Effectiveness of Self-Supervised Pre-Training for ASR2020
- Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model2019
- MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations2019