Chasing Random: Instruction Selection Strategies Fail to Generalize
Preprint
Abstract
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of methods that filter a large, noisy instruction-tuning datasets down to high-quality subset which works just as well. However, typically, the performance of these methods is not demonstrated across a uniform experimental setup and thus their generalization capabilities are not well established. In this work, we analyze popular selection strategies across different source datasets, selection budgets and evaluation benchmarks: Our results indicate that selection strategies generalize poorly, often failing to consistently outperform even random baselines. We also analyze the cost-performance trade-offs of using data selection. Our findings reveal that data selection can often exceed the cost of fine-tuning on the full dataset, yielding only marginal and sometimes no gains compared to tuning on the full dataset or a random subset.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- A Survey on Data Selection for LLM Instruction TuningJournal of Artificial Intelligence Research, 2025
- Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models2025
- Large Language Model Instruction Following: A Survey of Progresses and ChallengesComputational Linguistics, 2024
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning2024
- InFoBench: Evaluating Instruction Following Ability in Large Language Models2024
- Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models2024
- G-DIG: Towards Gradient-based DIverse and hiGh-quality Instruction Data Selection for Machine Translation2024
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning2024