Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark
Preprint
- 29 May 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Benchmark results for Large Language Models often show inconsistencies across different studies. This paper investigates the challenges of reproducing these results in automatic bugfixing using LLMs, on the HumanEvalFix benchmark. To determine the cause of the differing results in the literature, we attempted to reproduce a subset of them by evaluating 11 models in the DeepSeekCoder, CodeGemma, and CodeLlama model families, in different sizes and tunings. A total of 32 unique results were reported for these models across studies, of which we successfully reproduced 16. We identified several relevant factors that influence the results. Base models can be confused with their instruction-tuned variants, making their results better than expected. Incorrect prompt templates or generation length can decrease benchmark performance, as well as using 4-bit quantization. Using sampling instead of greedy decoding can increase variance, especially with higher temperature values. We found that precision and 8-bit quantization have less influence on benchmark results.Keywords
All Related Versions
- Version 1, 2025-05-29, Preprints
- Published version: Software, 4 (3), 17.
This publication has 7 references indexed in Scilit:
- Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review2024
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering2024
- WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning2024
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X2023
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation2023
- BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies2020
- QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge2017