Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning
Authors:
- Ming LiMLMing Li
- Shwai HeSHShwai He
- Tianyi ZhouTZTianyi Zhou
Preprint
Abstract
Recent advancements in Large Language Models (LLMs) have expanded the horizons of natural language understanding and generation. Notably, the output control and alignment with the input of LLMs can be refined through instruction tuning. However, as highlighted in several studies, low-quality data in the training set are usually detrimental to instruction tuning, resulting in inconsistent or even misleading LLM outputs. We propose a novel method, termed "reflection-tuning," which addresses the problem by self-improvement and judging capabilities of LLMs. This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data. Extensive experiments on widely used evaluation benchmarks show that LLMs trained with our recycled data outperform those trained with existing datasets in various benchmarks.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Automated Correction StrategiesTransactions of the Association for Computational Linguistics, 2024
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and BeyondACM Transactions on Knowledge Discovery From Data, 2024
- Large Language Models are not Fair Evaluators2024
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning2024
- Large Language Models Can Self-Improve2023
- Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data2023
- Koala: An Index for Quantifying Overlaps with Pre-training Corpora2023
- Cross-Task Generalization via Natural Language Crowdsourcing Instructions2022
- Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better ScienceTransactions of the Association for Computational Linguistics, 2018