Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
Preprint
Abstract
Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Differentiable Good Arm Identification2025
- A Study in Dataset Pruning for Image Super-Resolution2024
- RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Model2024
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning2024
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning2024
- Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023