CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
Authors:
- Hao YuHYHao Yu
- Bo ShenBSBo Shen
- Dezhi RanDRDezhi Ran
- Jiaxin ZhangJZJiaxin Zhang
- Qi ZhangQZQi Zhang
- Yuchi MaYMYuchi Ma
- Guangtai LiangGLGuangtai Liang
- Ying LiYLYing Li
- Qianxiang WangQWQianxiang Wang
- Tao XieTXTao Xie
Preprint
Abstract
Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.Keywords
All Related Versions
- Version 1, 2023-02-01, ArXiv
- Version 2, 2023-10-25, ArXiv
- Version 3, 2024-02-23, ArXiv
- Published version: , 1.
This publication has 6 references indexed in Scilit:
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code GenerationIEEE Transactions on Software Engineering, 2023
- Execution-Based Evaluation for Open-Domain Code Generation2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Mapping Language to Code in Programmatic Context2018