Emerging trends: A gentle introduction to fine-tuning
Open Access
- 26 October 2021
- journal article
- research article
- Published by Cambridge University Press (CUP) in Natural Language Engineering
- Vol. 27 (6) , 763-778
Abstract
The previous Emerging Trends article (Churchet al., 2021.Natural Language Engineering27(5), 631–645.) introduced deep nets to poets. Poets is an imperfect metaphor, intended as a gesture toward inclusion. The future for deep nets will benefit by reaching out to a broad audience of potential users, including people with little or no programming skills, and little interest in training models. That paper focused on inference, the use of pre-trained models, as is, without fine-tuning. The goal of this paper is to make fine-tuning more accessible to a broader audience. Since fine-tuning is more challenging than inference, the examples in this paper will require modest programming skills, as well as access to a GPU. Fine-tuning starts with a general purpose base (foundation) model and uses a small training set of labeled data to produce a model for a specific downstream application. There are many examples of fine-tuning in natural language processing (question answering (SQuAD) and GLUE benchmark), as well as vision and speech.Keywords
This publication has 43 references indexed in Scilit:
- The First Winograd Schema Challenge at IJCAI‐16AI Magazine, 2017
- Deep Residual Learning for Image Recognition2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Librispeech: An ASR corpus based on public domain audio books2015
- A Survey on Transfer LearningIEEE Transactions on Knowledge and Data Engineering, 2009
- Abbreviation definition identification based on automatic precision estimatesBMC Bioinformatics, 2008
- The PASCAL Recognising Textual Entailment Challenge2006
- Summarizing performance is no mean feat [computer performance analysis]2005
- Combining labeled and unlabeled data with co-training1998