LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
Authors:
- Yunhui XiaYXYunhui Xia
- Wei ShenWSWei Shen
- Yan WangYWYan Wang
- Jason Klein LiuJLJason Klein Liu
- Huifeng SunHSHuifeng Sun
- Siyue WuSWSiyue Wu
- Jian HuJHJian Hu
- Xiaolong XuXiaolong Xu
Preprint
Abstract
We introduce LeetCodeDataset, a high-quality benchmark for evaluating and training code-generation models, addressing two key challenges in LLM research: the lack of reasoning-focused coding benchmarks and self-contained training testbeds. By curating LeetCode Python problems with rich metadata, broad coverage, 100+ test cases per problem, and temporal splits (pre/post July 2024), our dataset enables contamination-free evaluation and efficient supervised fine-tuning (SFT). Experiments show reasoning models significantly outperform non-reasoning counterparts, while SFT with only 2.6K model-generated solutions achieves performance comparable to 110K-sample counterparts. The dataset and evaluation framework are available on Hugging Face and Github.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Leveraging Web-Crawled Data for High-Quality Fine-Tuning2024
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code GenerationIEEE Transactions on Software Engineering, 2023
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation2023