MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
Preprint
Abstract
While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a quantitative evaluation of performance on the MCoNaLa dataset by testing with state-of-the-art code generation systems. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesTransactions of the Association for Computational Linguistics, 2020
- An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code ExamplesIEEE Transactions on Software Engineering, 2020
- Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation2020
- A Transformer-based Approach for Source Code Summarization2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Learning to mine aligned code and natural language pairs from stack overflow2018
- StaQC2018
- Learning multilingual named entity recognition from WikipediaArtificial Intelligence, 2013