CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
Authors:
- Hao YuHYHao Yu
- Bo ShenBSBo Shen
- Dezhi RanDRDezhi Ran
- Jiaxin ZhangJZJiaxin Zhang
- Qi ZhangQZQi Zhang
- Yuchi MaYMYuchi Ma
- Guangtai LiangGLGuangtai Liang
- Ying LiYLYing Li
- Tao XieTXTao Xie
- Qianxiang WangQWQianxiang Wang
Preprint
Abstract
Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To validate the performance of these models, multiple existing benchmarks (e.g., AiXBench and HumanEval) are proposed, including only cases of generating a standalone function, i.e., a function that invokes or accesses only built-in functions and standard libraries. However, standalone functions constitute only about 30\% of functions from real open-source projects. To assess a model's performance for pragmatic code generation (i.e., code generation for real settings of open source or proprietary code), in this paper, we propose a benchmark named CoderEval of pragmatic code generation with generative pre-trained models. Compared with the widely-used HumanEval benchmark from OpenAI, CoderEval can be used to assess the performance of models against pragmatic code generation beyond just generating standalone functions. Through the evaluation of three public available models (CodeGen, PanGu-Coder, and Codex) on CoderEval, we analyze and discuss the current progress and future directions of pragmatic code generation with a generative pre-trained model.Keywords
All Related Versions
- Version 1, 2023-02-01, ArXiv
- Version 2, 2023-10-25, ArXiv
- Version 3, 2024-02-23, ArXiv
- Published version: , 1.
This publication has 7 references indexed in Scilit:
- Competition-level code generation with AlphaCodeScience, 2022
- GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Mapping Language to Code in Programmatic Context2018
- Discrete Signal Processing on Graphs: Sampling TheoryIEEE Transactions on Signal Processing, 2015
- Sampling large data on graphs2014
- BLEU2001