SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs
Authors:
- Changzhi ZhouCZChangzhi Zhou
- Linmei HuLinmei Hu
- Luhao ZhangLZLuhao Zhang
Preprint
Abstract
Existing code large language models (LLMs) often rely on large-scale instruction data distilled from proprietary LLMs for fine-tuning, which typically incurs high costs. In this paper, we explore the potential of small-scale open-source LLMs (e.g., 7B) as synthesizers for high-quality code instruction data construction. We first observe that the data synthesis capability of small-scale LLMs can be enhanced by training on a few superior data synthesis samples from proprietary LLMs. Building on this, we propose a novel iterative self-distillation approach to bootstrap small-scale LLMs, transforming them into powerful synthesizers that reduce reliance on proprietary LLMs and minimize costs. Concretely, in each iteration, to obtain diverse and high-quality self-distilled data, we design multi-checkpoint sampling and multi-aspect scoring strategies for initial data selection. Furthermore, to identify the most influential samples, we introduce a gradient-based influence estimation method for final data filtering. Based on the code instruction datasets from the small-scale synthesizers, we develop SCoder, a family of code generation models fine-tuned from DeepSeek-Coder. SCoder models achieve state-of-the-art code generation capabilities, demonstrating the effectiveness of our method.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Effective Tool Augmented Multi-Agent Framework for Data AnalysisData Intelligence, 2024
- Large Knowledge Model: Perspectives and ChallengesData Intelligence, 2024
- ChatGPT is a Remarkable Tool—For ExpertsData Intelligence, 2024
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data2024
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement2024
- Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges2024
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings2023