An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models
Authors:
Preprint
Abstract
Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.
Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- CiT: Curation in Training for Effective Vision-Language Data2023
- Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks2023
- Active Learning for Natural Language Generation2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor2023
- ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization2022
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions2022
- Lazier Than Lazy GreedyProceedings of the AAAI Conference on Artificial Intelligence, 2015
- Active learning with support vector machinesWIREs Data Mining and Knowledge Discovery, 2014
- Facility Location2009